2024-09-12T15:44:11.955 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-12T15:44:11.979 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T15:44:12.079 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: idryomov@gmail.com first_in_suite: false job_id: '7902659' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 1680 sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 sleep_before_teardown: 0 subset: 42/100 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOGlv8KdgRy99aYpxFzc9OxbFOhe+Psd2fGj+dWHhm0eXiEVM74bK2aUhC+j8nwkTndUeu+LbLlyO70xx6cAFLA= smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnPQM/fgXjYviHXSJUTvWuMdxI2P0i0UhacxnQunjFI8ccRd23NudHOCUn715FX1le15xAEncHuHd6ajf78PJI= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-12_15:21:08 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-12T15:44:12.080 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa; will attempt to use it 2024-09-12T15:44:12.080 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks 2024-09-12T15:44:12.081 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-12T15:44:12.082 INFO:teuthology.task.internal:Checking packages... 2024-09-12T15:44:12.107 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1' 2024-09-12T15:44:12.107 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-12T15:44:12.107 INFO:teuthology.packaging:ref: None 2024-09-12T15:44:12.108 INFO:teuthology.packaging:tag: None 2024-09-12T15:44:12.108 INFO:teuthology.packaging:branch: wip-dis-testing 2024-09-12T15:44:12.108 INFO:teuthology.packaging:sha1: d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T15:44:12.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing 2024-09-12T15:44:12.273 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4883.gd61b0886 2024-09-12T15:44:12.275 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-12T15:44:12.281 INFO:teuthology.task.internal:no buildpackages task found 2024-09-12T15:44:12.281 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-12T15:44:12.293 INFO:teuthology.task.internal:Saving configuration 2024-09-12T15:44:12.310 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-12T15:44:12.320 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-12T15:44:12.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 15:39:32.445428', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOGlv8KdgRy99aYpxFzc9OxbFOhe+Psd2fGj+dWHhm0eXiEVM74bK2aUhC+j8nwkTndUeu+LbLlyO70xx6cAFLA='} 2024-09-12T15:44:12.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-12 15:39:32.446980', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnPQM/fgXjYviHXSJUTvWuMdxI2P0i0UhacxnQunjFI8ccRd23NudHOCUn715FX1le15xAEncHuHd6ajf78PJI='} 2024-09-12T15:44:12.349 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-12T15:44:12.358 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-12T15:44:12.358 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-12T15:44:12.358 INFO:teuthology.run_tasks:Running task console_log... 2024-09-12T15:44:12.420 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7facaeb5bac0>, signals=[15]) 2024-09-12T15:44:12.420 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-12T15:44:12.430 INFO:teuthology.task.internal:Opening connections... 2024-09-12T15:44:12.430 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T15:44:12.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:44:12.546 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T15:44:12.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:44:12.623 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-12T15:44:12.630 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-09-12T15:44:12.649 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-09-12T15:44:12.649 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-12T15:44:12.708 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-09-12T15:44:12.709 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-09-12T15:44:12.709 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-12T15:44:12.709 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-09-12T15:44:12.709 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-12T15:44:12.709 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-12T15:44:12.709 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-12T15:44:12.710 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-09-12T15:44:12.728 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-09-12T15:44:12.748 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-09-12T15:44:12.748 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-09-12T15:44:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-09-12T15:44:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-09-12T15:44:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-12T15:44:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-12T15:44:12.808 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-09-12T15:44:12.828 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-12T15:44:12.839 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-12T15:44:12.866 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-12T15:44:12.867 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-09-12T15:44:12.869 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-09-12T15:44:12.886 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-12T15:44:12.893 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-12T15:44:12.893 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-09-12T15:44:12.927 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-09-12T15:44:12.953 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-12T15:44:13.000 INFO:teuthology.run_tasks:Running task kernel... 2024-09-12T15:44:13.019 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-12T15:44:13.020 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-12T15:44:13.020 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T15:44:13.020 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-12T15:44:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T15:44:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T15:44:13.038 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-12T15:44:13.039 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-09-12T15:44:13.095 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-503.el9.x86_64 2024-09-12T15:44:13.096 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-503.el9.x86_64 2024-09-12T15:44:13.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-09-12T15:44:13.097 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-503.el9.x86_64 2024-09-12T15:44:13.097 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-503.el9.x86_64 2024-09-12T15:44:13.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-09-12T15:44:15.396 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-12T15:44:16.191 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.2 MB 00:01 2024-09-12T15:44:19.426 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 20 MB 00:02 2024-09-12T15:44:23.785 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-12T15:44:24.647 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 20 MB 00:08 2024-09-12T15:44:26.035 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-12T15:44:26.814 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-12T15:44:29.014 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-12T15:44:30.557 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 19 kB 00:00 2024-09-12T15:44:31.411 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-12T15:44:32.571 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-12T15:44:34.669 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-12T15:44:34.669 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-12T15:44:34.766 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:44:34.775 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:44:34.775 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:44:34.776 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-09-12T15:44:34.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:44:34.780 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 79 M 2024-09-12T15:44:34.781 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 126 M 2024-09-12T15:44:34.781 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:44:34.988 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 254 kB/s | 23 kB 00:00 2024-09-12T15:44:37.133 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-12T15:44:37.472 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 6.1 MB/s | 16 MB 00:02 2024-09-12T15:44:39.165 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-12T15:44:39.165 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-12T15:44:39.262 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:44:39.277 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:44:39.277 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:44:39.278 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-09-12T15:44:39.279 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:44:39.279 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 79 M 2024-09-12T15:44:39.279 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 126 M 2024-09-12T15:44:39.279 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:44:39.414 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.5 MB/s | 29 MB 00:04 2024-09-12T15:44:39.443 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 195 kB/s | 23 kB 00:00 2024-09-12T15:44:40.647 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.0 MB/s | 35 MB 00:05 2024-09-12T15:44:40.647 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:44:40.647 INFO:teuthology.orchestra.run.smithi016.stdout:Total 13 MB/s | 79 MB 00:05 2024-09-12T15:44:41.364 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:44:41.748 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:44:41.749 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:44:42.135 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.6 MB/s | 16 MB 00:02 2024-09-12T15:44:42.731 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:44:42.731 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:44:43.135 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 7.7 MB/s | 29 MB 00:03 2024-09-12T15:44:44.885 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2024-09-12T15:44:44.885 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:44:44.886 INFO:teuthology.orchestra.run.smithi019.stdout:Total 14 MB/s | 79 MB 00:05 2024-09-12T15:44:45.324 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:44:45.603 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:44:45.972 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:44:45.972 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:44:45.982 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-12T15:44:46.064 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T15:44:46.965 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:44:46.965 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:44:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T15:44:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T15:44:49.556 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:44:50.230 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-12T15:44:50.311 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T15:44:53.116 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T15:44:53.239 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T15:44:54.140 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T15:44:54.345 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:44:58.360 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T15:44:58.398 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:44:59.486 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:03.534 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:39.025 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:39.025 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-12T15:45:39.025 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T15:45:39.025 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T15:45:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:45:39.869 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:45:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-12T15:45:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-12T15:45:39.869 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-12T15:45:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-12T15:45:39.870 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:45:39.870 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:45:40.228 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T15:45:40.313 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T15:45:40.313 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T15:45:40.313 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-12T15:45:40.314 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-12T15:45:40.314 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-12T15:45:40.314 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-12T15:45:40.314 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-09-12T15:45:40.314 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T15:45:40.373 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T15:45:40.374 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T15:45:40.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-09-12T15:45:40.405 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-09-12T15:45:40.405 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-09-12T15:45:40.405 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-09-12T15:45:40.405 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 82.el9 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8274670 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-09-12T15:45:40.406 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-09-12T15:45:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:45:40.407 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-12T15:45:40.407 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-12T15:45:40.407 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-12T15:45:40.407 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-09-12T15:45:40.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:45:40.407 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-09-12T15:45:40.408 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-12T15:45:40.408 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-505.el9.x86_64 2024-09-12T15:45:40.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-12T15:45:41.125 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-09-12T15:45:42.398 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:42.440 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:42.440 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-12T15:45:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-09-12T15:45:42.484 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-12T15:45:42.500 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.lowsTelqC3 2024-09-12T15:45:42.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lowsTelqC3 2024-09-12T15:45:42.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.lowsTelqC3 2024-09-12T15:45:42.708 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.lowsTelqC3 is 6KB 2024-09-12T15:45:42.756 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.lowsTelqC3 2024-09-12T15:45:42.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-09-12T15:45:42.783 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:42.784 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-12T15:45:42.784 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-12T15:45:42.784 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-12T15:45:42.834 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-12T15:45:42.834 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-12T15:45:42.834 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-12T15:45:42.834 INFO:teuthology.orchestra.run.smithi016.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-12T15:45:42.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-12T15:45:42.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-09-12T15:45:43.639 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:45:43.640 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:45:44.006 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T15:45:44.100 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T15:45:44.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T15:45:44.100 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-12T15:45:44.100 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-12T15:45:44.100 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-12T15:45:44.101 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-12T15:45:44.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2024-09-12T15:45:44.101 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-09-12T15:45:44.159 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-12T15:45:44.159 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-12T15:45:44.159 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2024-09-12T15:45:44.196 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 82.el9 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8274670 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-12T15:45:44.197 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:45:44.198 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2024-09-12T15:45:44.199 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-12T15:45:44.199 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-505.el9.x86_64 2024-09-12T15:45:44.200 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-12T15:45:44.900 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2024-09-12T15:45:46.148 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-12T15:45:46.164 INFO:teuthology.orchestra.run.smithi019.stderr:done 2024-09-12T15:45:46.166 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-09-12T15:45:46.182 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.EEYMPcIx0X 2024-09-12T15:45:46.182 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EEYMPcIx0X 2024-09-12T15:45:46.248 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.EEYMPcIx0X 2024-09-12T15:45:46.391 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.EEYMPcIx0X is 6KB 2024-09-12T15:45:46.439 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.EEYMPcIx0X 2024-09-12T15:45:46.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2024-09-12T15:45:46.526 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-12T15:45:46.526 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-12T15:45:46.526 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-12T15:45:46.526 INFO:teuthology.orchestra.run.smithi019.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-12T15:45:46.528 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-12T15:45:46.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2024-09-12T15:46:13.013 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-12T15:46:13.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T15:46:13.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:46:16.697 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-12T15:46:16.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-12T15:46:16.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:47:13.052 DEBUG:teuthology.orchestra.remote:timed out 2024-09-12T15:47:16.702 DEBUG:teuthology.orchestra.remote:timed out 2024-09-12T15:47:22.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T15:47:22.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:47:22.361 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-12T15:47:22.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T15:47:22.714 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-12T15:47:22.715 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-12T15:47:22.730 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-505.el9.x86_64 2024-09-12T15:47:22.730 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-12T15:47:22.730 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-12T15:47:22.730 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-12T15:47:23.731 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-12T15:47:23.731 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-12T15:47:23.835 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-09-12T15:47:23.872 DEBUG:teuthology.parallel:result is None 2024-09-12T15:47:25.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-12T15:47:25.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:47:26.112 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-09-12T15:47:26.650 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-12T15:47:26.650 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-12T15:47:26.650 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-09-12T15:47:26.667 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-505.el9.x86_64 2024-09-12T15:47:26.667 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-12T15:47:26.667 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-12T15:47:26.668 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-12T15:47:27.669 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-12T15:47:27.669 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-12T15:47:27.736 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-09-12T15:47:27.764 DEBUG:teuthology.parallel:result is None 2024-09-12T15:47:27.764 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-12T15:47:27.771 INFO:teuthology.task.internal:Creating test directory... 2024-09-12T15:47:27.771 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T15:47:27.774 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-12T15:47:27.789 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-12T15:47:27.796 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-12T15:47:27.806 INFO:teuthology.task.internal:Creating archive directory... 2024-09-12T15:47:27.806 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T15:47:27.834 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-12T15:47:27.933 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-12T15:47:27.942 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-12T15:47:27.943 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T15:47:27.945 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-12T15:47:27.980 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T15:47:27.982 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T15:47:27.991 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T15:47:27.994 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-12T15:47:27.995 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-12T15:47:28.004 INFO:teuthology.task.internal:Configuring sudo... 2024-09-12T15:47:28.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T15:47:28.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-12T15:47:28.068 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-12T15:47:28.082 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-12T15:47:28.082 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T15:47:28.112 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-12T15:47:28.128 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T15:47:28.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T15:47:28.267 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T15:47:28.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T15:47:28.397 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:47:28.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T15:47:28.467 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T15:47:28.506 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-12T15:47:28.563 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T15:47:28.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-12T15:47:28.687 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:47:28.687 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-12T15:47:28.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-09-12T15:47:28.757 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-09-12T15:47:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T15:47:28.845 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T15:47:29.229 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-12T15:47:29.236 INFO:teuthology.task.internal:Starting timer... 2024-09-12T15:47:29.237 INFO:teuthology.run_tasks:Running task pcp... 2024-09-12T15:47:29.247 INFO:teuthology.run_tasks:Running task selinux... 2024-09-12T15:47:29.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-09-12T15:47:29.318 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-09-12T15:47:29.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-09-12T15:47:29.376 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-09-12T15:47:29.377 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-12T15:47:29.378 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-09-12T15:47:29.396 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-09-12T15:47:29.396 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-09-12T15:47:29.419 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-09-12T15:47:29.419 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi019.front.sepia.ceph.com': 'permissive'} 2024-09-12T15:47:29.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T15:47:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T15:47:29.466 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T15:47:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T15:47:29.493 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-12T15:47:29.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-09-12T15:47:29.550 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-09-12T15:47:29.577 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-12T15:47:29.590 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-12T15:47:29.604 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-12T15:47:29.606 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi019.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-12T15:51:20.979 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi019.front.sepia.ceph.com')] 2024-09-12T15:51:20.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T15:51:20.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:51:21.062 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-12T15:51:21.146 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-12T15:51:21.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-12T15:51:21.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-12T15:51:21.227 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-09-12T15:51:21.309 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-09-12T15:51:21.310 INFO:teuthology.run_tasks:Running task clock... 2024-09-12T15:51:21.324 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-12T15:51:21.324 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T15:51:21.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T15:51:21.328 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-12T15:51:21.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T15:51:21.359 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-12T15:51:21.375 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-12T15:51:21.382 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-12T15:51:21.398 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-12T15:51:21.402 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-09-12T15:51:21.414 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-09-12T15:51:21.424 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-09-12T15:51:21.429 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-12T15:51:21.436 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-09-12T15:51:21.444 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-12T15:51:21.451 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-12T15:51:21.466 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-12T15:51:21.502 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-12T15:51:21.505 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T15:51:21.506 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-12T15:51:21.506 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.506 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.506 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.506 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.532 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-09-12T15:51:21.536 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T15:51:21.536 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-09-12T15:51:21.536 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.537 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.537 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.537 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T15:51:21.537 INFO:teuthology.run_tasks:Running task install... 2024-09-12T15:51:21.544 DEBUG:teuthology.task.install:project ceph 2024-09-12T15:51:21.545 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}} 2024-09-12T15:51:21.545 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'} 2024-09-12T15:51:21.545 INFO:teuthology.task.install:Using flavor: default 2024-09-12T15:51:21.552 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-12T15:51:21.552 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-12T15:51:21.553 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'tag': None, 'wait_for_package': False} 2024-09-12T15:51:21.553 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T15:51:21.556 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'tag': None, 'wait_for_package': False} 2024-09-12T15:51:21.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T15:51:21.703 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/ 2024-09-12T15:51:21.704 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4883.gd61b0886 2024-09-12T15:51:21.707 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/ 2024-09-12T15:51:21.708 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4883.gd61b0886 2024-09-12T15:51:21.816 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-12T15:51:21.816 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:51:21.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-12T15:51:21.820 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-dis-testing/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-12T15:51:21.820 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:51:21.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-12T15:51:21.843 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-12T15:51:21.844 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-12T15:51:21.849 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-12T15:51:21.849 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d61b08865d4d98738f45ff2e4806d7fb6d03eca1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-12T15:51:21.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-12T15:51:21.916 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-12T15:51:21.989 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-12T15:51:21.993 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-12T15:51:22.054 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-09-12T15:51:22.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-09-12T15:51:22.060 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-09-12T15:51:22.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-09-12T15:51:22.287 INFO:teuthology.orchestra.run.smithi016.stdout:45 files removed 2024-09-12T15:51:22.299 INFO:teuthology.orchestra.run.smithi019.stdout:45 files removed 2024-09-12T15:51:22.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw 2024-09-12T15:51:22.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2024-09-12T15:51:22.904 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 293 kB/s | 84 kB 00:00 2024-09-12T15:51:22.906 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 298 kB/s | 84 kB 00:00 2024-09-12T15:51:23.213 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 92 kB/s | 24 kB 00:00 2024-09-12T15:51:23.323 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 65 kB/s | 24 kB 00:00 2024-09-12T15:51:23.488 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-12T15:51:23.608 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-12T15:51:24.249 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-12T15:51:26.084 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-12T15:51:27.297 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.2 MB 00:03 2024-09-12T15:51:30.493 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-12T15:51:31.406 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 6.9 MB/s | 20 MB 00:02 2024-09-12T15:51:32.260 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 19 kB 00:00 2024-09-12T15:51:33.431 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-12T15:51:35.757 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-12T15:51:37.307 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-12T15:51:38.112 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-12T15:51:39.112 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-09-12T15:51:41.499 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 11 M 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.5 M 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T15:51:41.514 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 22 M 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 25 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 755 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 508 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T15:51:41.515 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 45 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 143 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 162 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 321 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 307 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 99 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-12T15:51:41.516 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout:Install 31 Packages 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-09-12T15:51:41.517 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:51:41.518 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 62 M 2024-09-12T15:51:41.518 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:51:42.080 INFO:teuthology.orchestra.run.smithi019.stdout:(1/33): ceph-base-19.3.0-4883.gd61b0886.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-12T15:51:42.138 INFO:teuthology.orchestra.run.smithi019.stdout:(2/33): ceph-selinux-19.3.0-4883.gd61b0886.el9. 431 kB/s | 25 kB 00:00 2024-09-12T15:51:42.222 INFO:teuthology.orchestra.run.smithi019.stdout:(3/33): libcephfs2-19.3.0-4883.gd61b0886.el9.x8 8.9 MB/s | 755 kB 00:00 2024-09-12T15:51:42.314 INFO:teuthology.orchestra.run.smithi019.stdout:(4/33): libradosstriper1-19.3.0-4883.gd61b0886. 5.4 MB/s | 508 kB 00:00 2024-09-12T15:51:42.431 INFO:teuthology.orchestra.run.smithi019.stdout:(5/33): ceph-radosgw-19.3.0-4883.gd61b0886.el9. 14 MB/s | 11 MB 00:00 2024-09-12T15:51:42.490 INFO:teuthology.orchestra.run.smithi019.stdout:(6/33): python3-ceph-argparse-19.3.0-4883.gd61b 776 kB/s | 45 kB 00:00 2024-09-12T15:51:42.598 INFO:teuthology.orchestra.run.smithi019.stdout:(7/33): librgw2-19.3.0-4883.gd61b0886.el9.x86_6 19 MB/s | 5.5 MB 00:00 2024-09-12T15:51:42.623 INFO:teuthology.orchestra.run.smithi019.stdout:(8/33): python3-ceph-common-19.3.0-4883.gd61b08 1.0 MB/s | 143 kB 00:00 2024-09-12T15:51:42.782 INFO:teuthology.orchestra.run.smithi019.stdout:(9/33): python3-cephfs-19.3.0-4883.gd61b0886.el 883 kB/s | 162 kB 00:00 2024-09-12T15:51:42.832 INFO:teuthology.orchestra.run.smithi019.stdout:(10/33): python3-rados-19.3.0-4883.gd61b0886.el 1.5 MB/s | 321 kB 00:00 2024-09-12T15:51:42.866 INFO:teuthology.orchestra.run.smithi019.stdout:(11/33): python3-rbd-19.3.0-4883.gd61b0886.el9. 3.6 MB/s | 307 kB 00:00 2024-09-12T15:51:42.891 INFO:teuthology.orchestra.run.smithi019.stdout:(12/33): python3-rgw-19.3.0-4883.gd61b0886.el9. 1.7 MB/s | 99 kB 00:00 2024-09-12T15:51:43.150 INFO:teuthology.orchestra.run.smithi019.stdout:(13/33): ceph-common-19.3.0-4883.gd61b0886.el9. 14 MB/s | 22 MB 00:01 2024-09-12T15:51:43.200 INFO:teuthology.orchestra.run.smithi019.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 320 kB/s | 107 kB 00:00 2024-09-12T15:51:43.250 INFO:teuthology.orchestra.run.smithi019.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-09-12T15:51:43.284 INFO:teuthology.orchestra.run.smithi019.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 434 kB/s | 171 kB 00:00 2024-09-12T15:51:43.510 INFO:teuthology.orchestra.run.smithi019.stdout:(17/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-09-12T15:51:43.535 INFO:teuthology.orchestra.run.smithi019.stdout:(18/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 417 kB/s | 160 kB 00:00 2024-09-12T15:51:43.560 INFO:teuthology.orchestra.run.smithi019.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-12T15:51:43.594 INFO:teuthology.orchestra.run.smithi019.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 503 kB/s | 42 kB 00:00 2024-09-12T15:51:43.619 INFO:teuthology.orchestra.run.smithi019.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 613 kB/s | 36 kB 00:00 2024-09-12T15:51:43.652 INFO:teuthology.orchestra.run.smithi019.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-09-12T15:51:43.686 INFO:teuthology.orchestra.run.smithi019.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-12T15:51:43.728 INFO:teuthology.orchestra.run.smithi019.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-12T15:51:43.753 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-09-12T15:51:43.754 INFO:teuthology.orchestra.run.smithi019.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-12T15:51:43.778 INFO:teuthology.orchestra.run.smithi019.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 308 kB/s | 26 kB 00:00 2024-09-12T15:51:43.804 INFO:teuthology.orchestra.run.smithi019.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-12T15:51:43.838 INFO:teuthology.orchestra.run.smithi019.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-12T15:51:43.904 INFO:teuthology.orchestra.run.smithi019.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-12T15:51:43.946 INFO:teuthology.orchestra.run.smithi019.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-12T15:51:44.030 INFO:teuthology.orchestra.run.smithi019.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-12T15:51:44.155 INFO:teuthology.orchestra.run.smithi019.stdout:(32/33): librados2-19.3.0-4883.gd61b0886.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-12T15:51:44.214 INFO:teuthology.orchestra.run.smithi019.stdout:(33/33): librbd1-19.3.0-4883.gd61b0886.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-12T15:51:44.219 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:51:44.219 INFO:teuthology.orchestra.run.smithi019.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-12T15:51:44.336 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:51:44.377 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:51:44.377 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:51:45.102 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:51:45.102 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:51:46.191 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:51:46.211 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:51:46.211 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:51:46.211 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:51:46.211 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 11 M 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.5 M 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 22 M 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 25 k 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-12T15:51:46.212 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 755 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 508 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 5.5 M 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 45 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 143 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 162 k 2024-09-12T15:51:46.213 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 321 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 307 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 99 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:51:46.214 INFO:teuthology.orchestra.run.smithi016.stdout:Install 31 Packages 2024-09-12T15:51:46.215 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-09-12T15:51:46.215 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:51:46.215 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 62 M 2024-09-12T15:51:46.215 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:51:46.708 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:51:46.799 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-12T15:51:46.860 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-12T15:51:46.894 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-12T15:51:46.917 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T15:51:46.940 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T15:51:47.128 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T15:51:47.181 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T15:51:47.379 INFO:teuthology.orchestra.run.smithi016.stdout:(1/33): ceph-base-19.3.0-4883.gd61b0886.el9.x86 8.6 MB/s | 5.5 MB 00:00 2024-09-12T15:51:47.513 INFO:teuthology.orchestra.run.smithi016.stdout:(2/33): ceph-radosgw-19.3.0-4883.gd61b0886.el9. 14 MB/s | 11 MB 00:00 2024-09-12T15:51:47.538 INFO:teuthology.orchestra.run.smithi016.stdout:(3/33): ceph-selinux-19.3.0-4883.gd61b0886.el9. 158 kB/s | 25 kB 00:00 2024-09-12T15:51:47.544 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-12T15:51:47.572 INFO:teuthology.orchestra.run.smithi016.stdout:(4/33): libcephfs2-19.3.0-4883.gd61b0886.el9.x8 13 MB/s | 755 kB 00:00 2024-09-12T15:51:47.605 INFO:teuthology.orchestra.run.smithi016.stdout:(5/33): libradosstriper1-19.3.0-4883.gd61b0886. 7.4 MB/s | 508 kB 00:00 2024-09-12T15:51:47.654 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T15:51:47.664 INFO:teuthology.orchestra.run.smithi016.stdout:(6/33): python3-ceph-argparse-19.3.0-4883.gd61b 771 kB/s | 45 kB 00:00 2024-09-12T15:51:47.748 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T15:51:47.756 INFO:teuthology.orchestra.run.smithi016.stdout:(7/33): python3-ceph-common-19.3.0-4883.gd61b08 1.5 MB/s | 143 kB 00:00 2024-09-12T15:51:47.837 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 8/35 2024-09-12T15:51:47.840 INFO:teuthology.orchestra.run.smithi016.stdout:(8/33): librgw2-19.3.0-4883.gd61b0886.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-12T15:51:47.863 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T15:51:47.865 INFO:teuthology.orchestra.run.smithi016.stdout:(9/33): python3-cephfs-19.3.0-4883.gd61b0886.el 1.5 MB/s | 162 kB 00:00 2024-09-12T15:51:47.890 INFO:teuthology.orchestra.run.smithi016.stdout:(10/33): python3-rados-19.3.0-4883.gd61b0886.el 6.3 MB/s | 321 kB 00:00 2024-09-12T15:51:47.924 INFO:teuthology.orchestra.run.smithi016.stdout:(11/33): python3-rbd-19.3.0-4883.gd61b0886.el9. 5.1 MB/s | 307 kB 00:00 2024-09-12T15:51:47.926 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T15:51:47.949 INFO:teuthology.orchestra.run.smithi016.stdout:(12/33): python3-rgw-19.3.0-4883.gd61b0886.el9. 1.7 MB/s | 99 kB 00:00 2024-09-12T15:51:47.963 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-12T15:51:48.007 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 11/35 2024-09-12T15:51:48.078 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T15:51:48.107 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T15:51:48.183 INFO:teuthology.orchestra.run.smithi016.stdout:(13/33): ceph-common-19.3.0-4883.gd61b0886.el9. 15 MB/s | 22 MB 00:01 2024-09-12T15:51:48.217 INFO:teuthology.orchestra.run.smithi016.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-09-12T15:51:48.300 INFO:teuthology.orchestra.run.smithi016.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2024-09-12T15:51:48.301 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T15:51:48.342 INFO:teuthology.orchestra.run.smithi016.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-12T15:51:48.376 INFO:teuthology.orchestra.run.smithi016.stdout:(17/33): libnbd-1.20.2-2.el9.x86_64.rpm 400 kB/s | 171 kB 00:00 2024-09-12T15:51:48.409 INFO:teuthology.orchestra.run.smithi016.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 662 kB 00:00 2024-09-12T15:51:48.460 INFO:teuthology.orchestra.run.smithi016.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 503 kB/s | 42 kB 00:00 2024-09-12T15:51:48.485 INFO:teuthology.orchestra.run.smithi016.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-12T15:51:48.488 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-12T15:51:48.510 INFO:teuthology.orchestra.run.smithi016.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-12T15:51:48.552 INFO:teuthology.orchestra.run.smithi016.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 390 kB/s | 36 kB 00:00 2024-09-12T15:51:48.578 INFO:teuthology.orchestra.run.smithi016.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 513 kB/s | 47 kB 00:00 2024-09-12T15:51:48.628 INFO:teuthology.orchestra.run.smithi016.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-12T15:51:48.653 INFO:teuthology.orchestra.run.smithi016.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-12T15:51:48.679 INFO:teuthology.orchestra.run.smithi016.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 281 kB/s | 26 kB 00:00 2024-09-12T15:51:48.679 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-12T15:51:48.704 INFO:teuthology.orchestra.run.smithi016.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-12T15:51:48.724 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-12T15:51:48.729 INFO:teuthology.orchestra.run.smithi016.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-12T15:51:48.765 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-12T15:51:48.797 INFO:teuthology.orchestra.run.smithi016.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-12T15:51:48.797 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-12T15:51:48.827 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-12T15:51:48.838 INFO:teuthology.orchestra.run.smithi016.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-12T15:51:48.859 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T15:51:48.896 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-12T15:51:48.922 INFO:teuthology.orchestra.run.smithi016.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-12T15:51:49.122 INFO:teuthology.orchestra.run.smithi016.stdout:(32/33): librados2-19.3.0-4883.gd61b0886.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-12T15:51:49.189 INFO:teuthology.orchestra.run.smithi016.stdout:(33/33): librbd1-19.3.0-4883.gd61b0886.el9.x86_ 9.3 MB/s | 3.2 MB 00:00 2024-09-12T15:51:49.194 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:51:49.194 INFO:teuthology.orchestra.run.smithi016.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-12T15:51:49.232 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-12T15:51:49.280 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T15:51:49.317 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:51:49.359 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:51:49.359 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:51:49.372 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T15:51:49.402 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 24/35 2024-09-12T15:51:49.438 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-12T15:51:49.955 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-12T15:51:49.991 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T15:51:50.055 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:51:50.055 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:51:50.063 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T15:51:50.144 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 28/35 2024-09-12T15:51:50.185 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 29/35 2024-09-12T15:51:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:51:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-12T15:51:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-12T15:51:51.770 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-12T15:51:51.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T15:51:51.823 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-12T15:51:52.020 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T15:51:52.074 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-12T15:51:52.444 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-12T15:51:52.555 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T15:51:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T15:51:52.681 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T15:51:52.715 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T15:51:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 8/35 2024-09-12T15:51:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T15:51:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 9/35 2024-09-12T15:51:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-12T15:51:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 11/35 2024-09-12T15:51:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T15:51:53.025 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T15:51:53.090 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 13/35 2024-09-12T15:51:53.125 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-12T15:51:53.235 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-12T15:51:53.268 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-12T15:51:53.304 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T15:51:53.317 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-12T15:51:53.343 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T15:51:53.357 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-12T15:51:53.387 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-12T15:51:53.425 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T15:51:53.436 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T15:51:53.436 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-12T15:51:53.436 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-12T15:51:53.437 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:51:53.466 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-12T15:51:53.469 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T15:51:53.814 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-12T15:51:53.858 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T15:51:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 23/35 2024-09-12T15:51:53.980 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 24/35 2024-09-12T15:51:54.016 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-12T15:51:54.536 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-12T15:51:54.569 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T15:51:54.667 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 27/35 2024-09-12T15:51:54.756 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 28/35 2024-09-12T15:51:54.797 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 29/35 2024-09-12T15:51:57.265 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T15:51:57.302 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T15:51:57.920 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 30/35 2024-09-12T15:51:57.964 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T15:51:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 31/35 2024-09-12T15:51:58.047 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-12T15:51:58.047 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-12T15:51:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:51:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T15:52:03.829 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T15:52:03.829 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-09-12T15:52:03.829 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-09-12T15:52:03.829 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-09-12T15:52:03.830 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-09-12T15:52:03.830 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-09-12T15:52:03.830 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-09-12T15:52:03.830 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-09-12T15:52:03.830 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:04.860 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T15:52:04.888 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T15:52:04.888 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:04.888 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-12T15:52:04.888 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T15:52:04.888 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T15:52:04.888 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:04.890 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T15:52:04.979 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T15:52:04.979 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 1/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 3/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 4/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 5/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 6/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T15:52:06.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 8/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 9/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 10/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 11/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 13/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-12T15:52:06.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-12T15:52:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-12T15:52:06.254 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-12T15:52:06.254 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T15:52:06.254 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-12T15:52:06.255 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 34/35 2024-09-12T15:52:07.291 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.292 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-12T15:52:07.293 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-12T15:52:07.294 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-12T15:52:07.295 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-12T15:52:07.295 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-20.el9.x86_64 2024-09-12T15:52:07.295 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-12T15:52:07.295 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:07.295 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:07.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2024-09-12T15:52:08.236 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:29 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:08.374 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:08.374 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:08.374 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:52:08.374 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:08.374 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 50 M 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 161 k 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-09-12T15:52:08.375 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:08.376 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 51 M 2024-09-12T15:52:08.376 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 217 M 2024-09-12T15:52:08.376 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:52:08.519 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-12T15:52:08.553 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-09-12T15:52:08.586 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:08.603 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): libcephsqlite-19.3.0-4883.gd61b0886.el9. 791 kB/s | 161 kB 00:00 2024-09-12T15:52:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T15:52:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 33/35 2024-09-12T15:52:09.646 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:09.646 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-12T15:52:09.646 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T15:52:09.646 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-12T15:52:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T15:52:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-12T15:52:09.746 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T15:52:09.945 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): ceph-test-19.3.0-4883.gd61b0886.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-12T15:52:09.947 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:09.947 INFO:teuthology.orchestra.run.smithi019.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-12T15:52:09.976 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:52:09.998 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:52:09.998 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:52:10.423 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:52:10.423 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:52:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T15:52:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 1/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 3/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 4/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 5/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_6 6/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 7/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9. 8/35 2024-09-12T15:52:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x8 9/35 2024-09-12T15:52:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 10/35 2024-09-12T15:52:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 11/35 2024-09-12T15:52:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 12/35 2024-09-12T15:52:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 13/35 2024-09-12T15:52:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-12T15:52:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-12T15:52:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-12T15:52:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-12T15:52:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-12T15:52:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-12T15:52:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-12T15:52:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 32/35 2024-09-12T15:52:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-12T15:52:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 34/35 2024-09-12T15:52:11.110 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:52:11.322 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-12T15:52:11.361 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-12T15:52:11.388 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-12T15:52:11.823 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-12T15:52:11.824 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:11.825 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-12T15:52:11.826 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-12T15:52:11.826 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-12T15:52:11.826 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-20.el9.x86_64 2024-09-12T15:52:11.826 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-12T15:52:11.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:11.826 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:12.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test 2024-09-12T15:52:12.781 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:29 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:12.919 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:12.919 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:12.919 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:52:12.919 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:12.919 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:52:12.919 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 50 M 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 161 k 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-09-12T15:52:12.920 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:12.921 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 51 M 2024-09-12T15:52:12.921 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 217 M 2024-09-12T15:52:12.921 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:52:13.262 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): libcephsqlite-19.3.0-4883.gd61b0886.el9. 741 kB/s | 161 kB 00:00 2024-09-12T15:52:13.287 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-12T15:52:13.413 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 424 kB/s | 64 kB 00:00 2024-09-12T15:52:14.613 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): ceph-test-19.3.0-4883.gd61b0886.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-12T15:52:14.615 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:14.615 INFO:teuthology.orchestra.run.smithi016.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-12T15:52:14.644 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:52:14.667 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:52:14.667 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:52:15.082 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:52:15.083 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:52:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:52:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-12T15:52:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-12T15:52:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T15:52:16.415 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T15:52:16.498 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T15:52:18.855 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T15:52:18.855 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 1/4 2024-09-12T15:52:18.855 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2/4 2024-09-12T15:52:18.855 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:19.164 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:19.478 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2024-09-12T15:52:20.078 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:41 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:20.221 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:20.223 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:20.223 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:52:20.223 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:20.223 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 6.6 k 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 2.4 M 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 1.5 M 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 291 k 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 4.8 M 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 17 M 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-12T15:52:20.224 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-12T15:52:20.225 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-12T15:52:20.226 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-12T15:52:20.227 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-12T15:52:20.227 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:20.227 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:52:20.227 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:20.227 INFO:teuthology.orchestra.run.smithi019.stdout:Install 37 Packages 2024-09-12T15:52:20.227 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:20.227 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 M 2024-09-12T15:52:20.228 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 109 M 2024-09-12T15:52:20.228 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:52:20.874 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 3/4 2024-09-12T15:52:20.949 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T15:52:21.446 INFO:teuthology.orchestra.run.smithi019.stdout:(1/37): ceph-19.3.0-4883.gd61b0886.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-09-12T15:52:21.979 INFO:teuthology.orchestra.run.smithi019.stdout:(2/37): ceph-mgr-19.3.0-4883.gd61b0886.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-09-12T15:52:22.113 INFO:teuthology.orchestra.run.smithi019.stdout:(3/37): ceph-mds-19.3.0-4883.gd61b0886.el9.x86_ 2.6 MB/s | 2.4 MB 00:00 2024-09-12T15:52:22.180 INFO:teuthology.orchestra.run.smithi019.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4883.gd61b 4.3 MB/s | 291 kB 00:00 2024-09-12T15:52:22.314 INFO:teuthology.orchestra.run.smithi019.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-12T15:52:22.597 INFO:teuthology.orchestra.run.smithi019.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 4.3 MB/s | 1.2 MB 00:00 2024-09-12T15:52:22.664 INFO:teuthology.orchestra.run.smithi019.stdout:(7/37): ceph-mon-19.3.0-4883.gd61b0886.el9.x86_ 3.9 MB/s | 4.8 MB 00:01 2024-09-12T15:52:22.689 INFO:teuthology.orchestra.run.smithi019.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-09-12T15:52:22.748 INFO:teuthology.orchestra.run.smithi019.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-12T15:52:22.773 INFO:teuthology.orchestra.run.smithi019.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 1.2 MB/s | 135 kB 00:00 2024-09-12T15:52:22.982 INFO:teuthology.orchestra.run.smithi019.stdout:(11/37): ceph-osd-19.3.0-4883.gd61b0886.el9.x86 17 MB/s | 17 MB 00:01 2024-09-12T15:52:23.007 INFO:teuthology.orchestra.run.smithi019.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 829 kB/s | 215 kB 00:00 2024-09-12T15:52:23.049 INFO:teuthology.orchestra.run.smithi019.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 684 kB/s | 188 kB 00:00 2024-09-12T15:52:23.091 INFO:teuthology.orchestra.run.smithi019.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-12T15:52:23.116 INFO:teuthology.orchestra.run.smithi019.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-12T15:52:23.150 INFO:teuthology.orchestra.run.smithi019.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 867 kB/s | 51 kB 00:00 2024-09-12T15:52:23.167 INFO:teuthology.orchestra.run.smithi019.stdout:(17/37): python3-mako-1.1.4-6.el9.noarch.rpm 932 kB/s | 172 kB 00:00 2024-09-12T15:52:23.193 INFO:teuthology.orchestra.run.smithi019.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 549 kB/s | 42 kB 00:00 2024-09-12T15:52:23.234 INFO:teuthology.orchestra.run.smithi019.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-12T15:52:23.260 INFO:teuthology.orchestra.run.smithi019.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 650 kB/s | 43 kB 00:00 2024-09-12T15:52:23.285 INFO:teuthology.orchestra.run.smithi019.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.5 MB/s | 172 kB 00:00 2024-09-12T15:52:23.310 INFO:teuthology.orchestra.run.smithi019.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-12T15:52:23.335 INFO:teuthology.orchestra.run.smithi019.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-12T15:52:23.364 INFO:teuthology.orchestra.run.smithi019.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 825 kB/s | 23 kB 00:00 2024-09-12T15:52:23.388 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 4/4 2024-09-12T15:52:23.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 1/4 2024-09-12T15:52:23.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2/4 2024-09-12T15:52:23.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-12T15:52:23.389 INFO:teuthology.orchestra.run.smithi019.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 778 kB/s | 19 kB 00:00 2024-09-12T15:52:23.414 INFO:teuthology.orchestra.run.smithi019.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-09-12T15:52:23.465 INFO:teuthology.orchestra.run.smithi019.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-12T15:52:23.490 INFO:teuthology.orchestra.run.smithi019.stdout:(28/37): lua-devel-5.4.4-4.el9.x86_64.rpm 66 kB/s | 22 kB 00:00 2024-09-12T15:52:23.515 INFO:teuthology.orchestra.run.smithi019.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-09-12T15:52:23.549 INFO:teuthology.orchestra.run.smithi019.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-09-12T15:52:23.574 INFO:teuthology.orchestra.run.smithi019.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 330 kB/s | 16 kB 00:00 2024-09-12T15:52:23.599 INFO:teuthology.orchestra.run.smithi019.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-12T15:52:23.625 INFO:teuthology.orchestra.run.smithi019.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-12T15:52:23.650 INFO:teuthology.orchestra.run.smithi019.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-12T15:52:23.676 INFO:teuthology.orchestra.run.smithi019.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-12T15:52:23.701 INFO:teuthology.orchestra.run.smithi019.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-12T15:52:23.726 INFO:teuthology.orchestra.run.smithi019.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-12T15:52:23.730 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:23.731 INFO:teuthology.orchestra.run.smithi019.stdout:Total 8.8 MB/s | 31 MB 00:03 2024-09-12T15:52:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-12T15:52:23.743 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:23.744 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:52:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-12T15:52:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-12T15:52:23.744 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:23.744 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:23.887 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:52:23.931 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:52:23.931 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:52:24.016 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph 2024-09-12T15:52:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:52:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:52:24.613 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:41 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:24.757 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:24.759 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 6.6 k 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 2.4 M 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 1.5 M 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 291 k 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 4.8 M 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 17 M 2024-09-12T15:52:24.760 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-12T15:52:24.761 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-12T15:52:24.762 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-12T15:52:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:24.764 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:52:24.764 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:24.764 INFO:teuthology.orchestra.run.smithi016.stdout:Install 37 Packages 2024-09-12T15:52:24.764 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:24.764 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 31 M 2024-09-12T15:52:24.764 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 109 M 2024-09-12T15:52:24.765 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:52:25.398 INFO:teuthology.orchestra.run.smithi016.stdout:(1/37): ceph-19.3.0-4883.gd61b0886.el9.x86_64.r 54 kB/s | 6.6 kB 00:00 2024-09-12T15:52:25.574 INFO:teuthology.orchestra.run.smithi016.stdout:(2/37): ceph-mgr-19.3.0-4883.gd61b0886.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-12T15:52:25.649 INFO:teuthology.orchestra.run.smithi016.stdout:(3/37): ceph-mds-19.3.0-4883.gd61b0886.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-12T15:52:25.758 INFO:teuthology.orchestra.run.smithi016.stdout:(4/37): ceph-mon-19.3.0-4883.gd61b0886.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-12T15:52:25.783 INFO:teuthology.orchestra.run.smithi016.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4883.gd61b 2.1 MB/s | 291 kB 00:00 2024-09-12T15:52:25.939 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:52:25.942 INFO:teuthology.orchestra.run.smithi016.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-12T15:52:25.992 INFO:teuthology.orchestra.run.smithi016.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-09-12T15:52:26.004 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-12T15:52:26.017 INFO:teuthology.orchestra.run.smithi016.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-12T15:52:26.037 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-12T15:52:26.065 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-12T15:52:26.141 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-12T15:52:26.193 INFO:teuthology.orchestra.run.smithi016.stdout:(9/37): ceph-osd-19.3.0-4883.gd61b0886.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-12T15:52:26.218 INFO:teuthology.orchestra.run.smithi016.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 623 kB/s | 125 kB 00:00 2024-09-12T15:52:26.247 INFO:teuthology.orchestra.run.smithi016.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 531 kB/s | 135 kB 00:00 2024-09-12T15:52:26.305 INFO:teuthology.orchestra.run.smithi016.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 2.1 MB/s | 188 kB 00:00 2024-09-12T15:52:26.306 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-12T15:52:26.347 INFO:teuthology.orchestra.run.smithi016.stdout:(13/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.4 MB/s | 215 kB 00:00 2024-09-12T15:52:26.348 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-12T15:52:26.414 INFO:teuthology.orchestra.run.smithi016.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-12T15:52:26.439 INFO:teuthology.orchestra.run.smithi016.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 838 kB/s | 77 kB 00:00 2024-09-12T15:52:26.444 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-12T15:52:26.465 INFO:teuthology.orchestra.run.smithi016.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 790 kB/s | 172 kB 00:00 2024-09-12T15:52:26.490 INFO:teuthology.orchestra.run.smithi016.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-12T15:52:26.515 INFO:teuthology.orchestra.run.smithi016.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 550 kB/s | 42 kB 00:00 2024-09-12T15:52:26.529 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-12T15:52:26.566 INFO:teuthology.orchestra.run.smithi016.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-12T15:52:26.575 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-12T15:52:26.591 INFO:teuthology.orchestra.run.smithi016.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.3 MB/s | 172 kB 00:00 2024-09-12T15:52:26.610 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-12T15:52:26.616 INFO:teuthology.orchestra.run.smithi016.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-12T15:52:26.639 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-12T15:52:26.642 INFO:teuthology.orchestra.run.smithi016.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-12T15:52:26.667 INFO:teuthology.orchestra.run.smithi016.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 711 kB/s | 18 kB 00:00 2024-09-12T15:52:26.682 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-12T15:52:26.692 INFO:teuthology.orchestra.run.smithi016.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-12T15:52:26.715 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-12T15:52:26.726 INFO:teuthology.orchestra.run.smithi016.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-09-12T15:52:26.751 INFO:teuthology.orchestra.run.smithi016.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-12T15:52:26.764 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-12T15:52:26.776 INFO:teuthology.orchestra.run.smithi016.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-12T15:52:26.808 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-12T15:52:26.810 INFO:teuthology.orchestra.run.smithi016.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 786 kB/s | 46 kB 00:00 2024-09-12T15:52:26.835 INFO:teuthology.orchestra.run.smithi016.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-12T15:52:26.842 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-12T15:52:26.853 INFO:teuthology.orchestra.run.smithi016.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 986 kB/s | 16 kB 00:00 2024-09-12T15:52:26.878 INFO:teuthology.orchestra.run.smithi016.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-09-12T15:52:26.891 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-12T15:52:26.903 INFO:teuthology.orchestra.run.smithi016.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-12T15:52:26.920 INFO:teuthology.orchestra.run.smithi016.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-09-12T15:52:26.931 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-12T15:52:26.945 INFO:teuthology.orchestra.run.smithi016.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-09-12T15:52:26.971 INFO:teuthology.orchestra.run.smithi016.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-12T15:52:26.996 INFO:teuthology.orchestra.run.smithi016.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-12T15:52:27.017 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-12T15:52:27.021 INFO:teuthology.orchestra.run.smithi016.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 40 kB/s | 22 kB 00:00 2024-09-12T15:52:27.026 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:27.026 INFO:teuthology.orchestra.run.smithi016.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-12T15:52:27.170 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-12T15:52:27.182 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:52:27.226 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:52:27.226 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:52:27.271 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-12T15:52:27.445 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-12T15:52:27.500 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-12T15:52:27.559 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-12T15:52:27.622 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-12T15:52:27.760 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:52:27.761 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:52:27.823 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-12T15:52:27.962 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-12T15:52:28.043 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-12T15:52:28.150 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-12T15:52:28.194 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-12T15:52:28.329 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-12T15:52:28.566 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 32/37 2024-09-12T15:52:28.596 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T15:52:28.624 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T15:52:28.624 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:28.624 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-12T15:52:28.624 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T15:52:28.625 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T15:52:28.625 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:52:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-12T15:52:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-12T15:52:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-12T15:52:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-12T15:52:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-12T15:52:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-12T15:52:29.555 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-12T15:52:29.633 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-12T15:52:29.680 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-12T15:52:29.714 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-12T15:52:29.743 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-12T15:52:29.779 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-12T15:52:29.820 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-12T15:52:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-12T15:52:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-12T15:52:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-12T15:52:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-12T15:52:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-12T15:52:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-12T15:52:30.180 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T15:52:30.209 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T15:52:30.209 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:30.209 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-12T15:52:30.209 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T15:52:30.209 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T15:52:30.209 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:30.278 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-12T15:52:30.367 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-12T15:52:30.428 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-12T15:52:30.464 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-12T15:52:30.523 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-12T15:52:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-12T15:52:30.689 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T15:52:30.714 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T15:52:30.714 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:30.714 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-12T15:52:30.715 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T15:52:30.715 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T15:52:30.715 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:30.795 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-12T15:52:30.844 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-12T15:52:30.925 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-12T15:52:30.964 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T15:52:30.990 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T15:52:30.990 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:30.990 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-12T15:52:30.990 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T15:52:30.990 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T15:52:30.990 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:31.023 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-12T15:52:31.047 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T15:52:31.059 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-12T15:52:31.205 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-12T15:52:31.381 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 32/37 2024-09-12T15:52:31.411 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T15:52:31.439 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 33/37 2024-09-12T15:52:31.439 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:31.439 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-12T15:52:31.439 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T15:52:31.439 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-12T15:52:31.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:32.809 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 1/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 3/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 4/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 5/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 6/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-12T15:52:32.810 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-12T15:52:32.811 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-12T15:52:32.812 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-12T15:52:32.812 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-12T15:52:32.812 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-12T15:52:32.813 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-12T15:52:32.987 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T15:52:33.017 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 34/37 2024-09-12T15:52:33.017 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:33.017 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-12T15:52:33.017 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T15:52:33.017 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-12T15:52:33.017 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T15:52:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 35/37 2024-09-12T15:52:33.514 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:33.514 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-12T15:52:33.514 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T15:52:33.514 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-12T15:52:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:33.653 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-12T15:52:33.654 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-12T15:52:33.655 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-12T15:52:33.656 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-12T15:52:33.657 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-12T15:52:33.657 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-12T15:52:33.657 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:33.657 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T15:52:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 36/37 2024-09-12T15:52:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-12T15:52:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T15:52:33.781 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-12T15:52:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:33.847 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T15:52:33.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2024-09-12T15:52:34.606 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:55 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:34.700 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:52:34.745 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:34.746 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:52:34.746 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:34.815 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2024-09-12T15:52:35.404 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:56 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:35.542 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:35.543 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:35.543 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:52:35.543 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:35.543 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:52:35.543 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 784 k 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 784 k 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 791 k 2024-09-12T15:52:35.544 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:52:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 37/37 2024-09-12T15:52:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 1/37 2024-09-12T15:52:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2/37 2024-09-12T15:52:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 3/37 2024-09-12T15:52:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 4/37 2024-09-12T15:52:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 5/37 2024-09-12T15:52:35.773 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9. 6/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-12T15:52:35.774 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-12T15:52:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-12T15:52:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-12T15:52:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-12T15:52:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-12T15:52:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-12T15:52:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-12T15:52:35.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-12T15:52:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-12T15:52:35.776 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-12T15:52:35.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-12T15:52:35.805 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-19.3.0-4883.gd61b0886.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-12T15:52:35.805 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:35.806 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-12T15:52:35.806 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:52:35.811 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:52:35.812 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:52:35.818 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:52:35.818 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:52:35.895 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:52:36.368 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:36.470 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:36.794 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-09-12T15:52:36.795 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-12T15:52:36.796 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-12T15:52:36.797 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-12T15:52:36.798 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-12T15:52:36.798 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-12T15:52:36.798 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-12T15:52:36.798 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-12T15:52:36.798 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:36.798 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:37.077 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:37.078 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:52:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:37.078 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:37.078 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:37.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base 2024-09-12T15:52:37.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2024-09-12T15:52:37.796 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:54 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:37.890 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:52:37.896 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:58 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:37.935 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:37.936 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:52:37.936 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:37.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install cephadm 2024-09-12T15:52:38.035 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:38.036 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:38.036 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-12T15:52:38.036 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:38.036 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:52:38.036 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 145 k 2024-09-12T15:52:38.036 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:38.037 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:52:38.037 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:38.037 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:52:38.037 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:38.037 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 145 k 2024-09-12T15:52:38.037 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 432 k 2024-09-12T15:52:38.037 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:52:38.244 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-19.3.0-4883.gd61b08 699 kB/s | 145 kB 00:00 2024-09-12T15:52:38.245 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:38.245 INFO:teuthology.orchestra.run.smithi019.stdout:Total 695 kB/s | 145 kB 00:00 2024-09-12T15:52:38.245 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:52:38.253 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:52:38.253 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:52:38.313 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:52:38.313 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:52:38.458 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:52:38.504 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T15:52:38.529 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T15:52:38.529 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:38.530 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-12T15:52:38.530 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:38.586 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:55 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:38.724 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 784 k 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:52:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:38.726 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 784 k 2024-09-12T15:52:38.726 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 791 k 2024-09-12T15:52:38.726 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:52:39.014 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm-19.3.0-4883.gd61b0886.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-12T15:52:39.014 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:39.015 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-12T15:52:39.015 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:52:39.020 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:52:39.020 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:52:39.027 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:52:39.027 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:52:39.099 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:52:39.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T15:52:39.454 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:39.454 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:52:39.454 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:39.454 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:39.454 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:39.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2024-09-12T15:52:39.962 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:52:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:40.235 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:52:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:40.235 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:40.236 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:40.286 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:01 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:40.379 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:52:40.424 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:40.425 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:52:40.425 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:40.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache 2024-09-12T15:52:40.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2024-09-12T15:52:41.034 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:58 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:41.073 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:02 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:41.175 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:41.175 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-12T15:52:41.175 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:41.175 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:52:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 145 k 2024-09-12T15:52:41.175 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:41.176 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:52:41.176 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:41.176 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:52:41.176 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:41.176 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 145 k 2024-09-12T15:52:41.176 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 432 k 2024-09-12T15:52:41.177 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:52:41.210 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:41.211 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:41.211 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:52:41.211 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 14 M 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 34 k 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 16 k 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-12T15:52:41.212 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-09-12T15:52:41.213 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:41.214 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 19 M 2024-09-12T15:52:41.214 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 159 M 2024-09-12T15:52:41.214 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:52:41.381 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-19.3.0-4883.gd61b08 712 kB/s | 145 kB 00:00 2024-09-12T15:52:41.381 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:41.381 INFO:teuthology.orchestra.run.smithi016.stdout:Total 706 kB/s | 145 kB 00:00 2024-09-12T15:52:41.381 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:52:41.389 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:52:41.389 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:52:41.450 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:52:41.450 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:52:41.598 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:52:41.665 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T15:52:41.692 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T15:52:41.692 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:52:41.692 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-12T15:52:41.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:41.898 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4883.gd6 222 kB/s | 34 kB 00:00 2024-09-12T15:52:41.923 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4883.gd61 92 kB/s | 16 kB 00:00 2024-09-12T15:52:42.056 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-09-12T15:52:42.107 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-09-12T15:52:42.165 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-12T15:52:42.182 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-12T15:52:42.258 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-12T15:52:42.283 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-12T15:52:42.300 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-12T15:52:42.325 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.1 MB/s | 863 kB 00:00 2024-09-12T15:52:42.350 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-12T15:52:42.509 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4883.gd61b08 19 MB/s | 14 MB 00:00 2024-09-12T15:52:42.512 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:42.513 INFO:teuthology.orchestra.run.smithi019.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-12T15:52:42.563 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:52:42.584 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:52:42.584 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:52:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886. 1/1 2024-09-12T15:52:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:42.668 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:52:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:52:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:42.668 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:42.879 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:52:42.880 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:52:42.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr 2024-09-12T15:52:43.491 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:00 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:43.496 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:52:43.569 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-12T15:52:43.583 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-mgr-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:52:43.603 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-12T15:52:43.628 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:43.629 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:52:43.629 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:43.651 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-12T15:52:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-12T15:52:43.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-dashboard 2024-09-12T15:52:43.778 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-12T15:52:43.930 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-12T15:52:44.024 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-12T15:52:44.069 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-12T15:52:44.097 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-12T15:52:44.129 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 10/12 2024-09-12T15:52:44.274 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:01 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:44.411 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 14 M 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 34 k 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 16 k 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-12T15:52:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-09-12T15:52:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:44.415 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 19 M 2024-09-12T15:52:44.415 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 159 M 2024-09-12T15:52:44.415 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:52:44.935 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4883.gd6 224 kB/s | 34 kB 00:00 2024-09-12T15:52:44.960 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4883.gd61 92 kB/s | 16 kB 00:00 2024-09-12T15:52:45.160 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-12T15:52:45.261 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-09-12T15:52:45.303 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.9 MB/s | 863 kB 00:00 2024-09-12T15:52:45.361 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.3 MB/s | 548 kB 00:00 2024-09-12T15:52:45.386 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 239 kB/s | 19 kB 00:00 2024-09-12T15:52:45.412 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-12T15:52:45.437 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-12T15:52:45.462 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-12T15:52:45.629 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4883.gd61b08 17 MB/s | 14 MB 00:00 2024-09-12T15:52:45.688 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.2 MB/s | 2.0 MB 00:00 2024-09-12T15:52:45.692 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:45.692 INFO:teuthology.orchestra.run.smithi016.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-12T15:52:45.744 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:52:45.766 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:52:45.766 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:52:46.065 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:52:46.069 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:52:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:52:46.873 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-12T15:52:46.916 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-12T15:52:46.963 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-12T15:52:46.993 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-12T15:52:47.088 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-12T15:52:47.225 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-12T15:52:47.325 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-12T15:52:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-12T15:52:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-12T15:52:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 10/12 2024-09-12T15:52:47.490 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 11/12 2024-09-12T15:52:47.549 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T15:52:49.447 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 1/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 2/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 3/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-12T15:52:49.448 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-12T15:52:49.786 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-12T15:52:49.786 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:49.786 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:52:49.786 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:49.787 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:52:50.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-12T15:52:50.614 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:11 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:52:50.755 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:52:50.756 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-09-12T15:52:50.756 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:52:50.756 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-09-12T15:52:50.756 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:52:50.756 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 7.4 M 2024-09-12T15:52:50.756 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:52:50.756 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:50.757 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:52:50.758 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-09-12T15:52:50.758 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-09-12T15:52:50.758 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:52:50.758 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 38 M 2024-09-12T15:52:50.758 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 212 M 2024-09-12T15:52:50.758 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:52:50.826 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 11/12 2024-09-12T15:52:50.888 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T15:52:51.323 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-48 17 MB/s | 7.4 MB 00:00 2024-09-12T15:52:51.398 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 438 kB/s | 33 kB 00:00 2024-09-12T15:52:51.432 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 345 kB/s | 191 kB 00:00 2024-09-12T15:52:51.507 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 243 kB/s | 18 kB 00:00 2024-09-12T15:52:51.592 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-09-12T15:52:51.617 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 352 kB/s | 38 kB 00:00 2024-09-12T15:52:51.717 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 39 MB/s | 4.9 MB 00:00 2024-09-12T15:52:51.742 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-12T15:52:51.768 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): libgfortran-11.5.0-2.el9.x86_64.rpm 901 kB/s | 801 kB 00:00 2024-09-12T15:52:51.793 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-09-12T15:52:51.901 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-09-12T15:52:52.243 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-12T15:52:52.244 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:52.244 INFO:teuthology.orchestra.run.smithi019.stdout:Total 26 MB/s | 38 MB 00:01 2024-09-12T15:52:52.608 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:52:52.635 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:52:52.635 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:52:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 12/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el 1/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noa 2/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9 3/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-12T15:52:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-12T15:52:53.084 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:52:53.084 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-12T15:52:53.201 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:53.202 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:52:53.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-12T15:52:53.871 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:52:53.954 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-12T15:52:53.995 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-12T15:52:54.049 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:11 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:52:54.097 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-12T15:52:54.135 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-12T15:52:54.188 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:52:54.189 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-09-12T15:52:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:52:54.189 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-09-12T15:52:54.189 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:52:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 7.4 M 2024-09-12T15:52:54.189 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:52:54.189 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-09-12T15:52:54.190 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-09-12T15:52:54.191 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:52:54.191 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 38 M 2024-09-12T15:52:54.191 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 212 M 2024-09-12T15:52:54.191 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:52:54.329 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-12T15:52:54.359 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-12T15:52:54.446 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-12T15:52:54.470 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-12T15:52:54.513 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.4 MB/s | 801 kB 00:00 2024-09-12T15:52:54.605 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 33 kB 00:00 2024-09-12T15:52:54.655 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-12T15:52:54.705 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-12T15:52:54.814 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-48 15 MB/s | 7.4 MB 00:00 2024-09-12T15:52:55.015 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-12T15:52:55.137 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-12T15:52:55.252 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-12T15:52:55.306 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-09-12T15:52:55.523 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.1 MB/s | 250 kB 00:00 2024-09-12T15:52:55.882 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.2 MB/s | 4.9 MB 00:01 2024-09-12T15:52:56.570 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T15:52:56.624 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.2 MB/s | 5.1 MB 00:01 2024-09-12T15:52:57.665 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-12T15:52:57.683 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T15:52:58.590 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.3 MB/s | 16 MB 00:03 2024-09-12T15:52:58.591 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:52:58.592 INFO:teuthology.orchestra.run.smithi016.stdout:Total 8.7 MB/s | 38 MB 00:04 2024-09-12T15:52:58.965 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:52:58.992 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:52:58.992 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:52:59.443 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:52:59.444 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T15:53:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 1/12 2024-09-12T15:53:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-12T15:53:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-12T15:53:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-12T15:53:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-12T15:53:00.184 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-12T15:53:00.185 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-12T15:53:00.185 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-12T15:53:00.185 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-12T15:53:00.185 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T15:53:00.185 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-12T15:53:00.234 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:00.306 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-12T15:53:00.357 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-12T15:53:00.460 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-12T15:53:00.499 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-12T15:53:00.609 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-12T15:53:00.610 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-12T15:53:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-12T15:53:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-12T15:53:00.611 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:00.611 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-12T15:53:00.839 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-12T15:53:00.917 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-12T15:53:00.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2024-09-12T15:53:01.590 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:22 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-12T15:53:01.722 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-12T15:53:01.730 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:01.731 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:01.731 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:53:01.731 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:01.731 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:01.731 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 49 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-12T15:53:01.732 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-12T15:53:01.733 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:01.733 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:01.733 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:01.733 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2024-09-12T15:53:01.733 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:01.733 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.2 M 2024-09-12T15:53:01.734 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-09-12T15:53:01.734 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:02.108 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): ceph-mgr-rook-19.3.0-4883.gd61b0886.el9 298 kB/s | 49 kB 00:00 2024-09-12T15:53:02.483 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 35 kB/s | 19 kB 00:00 2024-09-12T15:53:02.525 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 44 kB/s | 26 kB 00:00 2024-09-12T15:53:02.867 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 293 kB/s | 222 kB 00:00 2024-09-12T15:53:03.037 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T15:53:03.209 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 219 kB/s | 159 kB 00:00 2024-09-12T15:53:03.242 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 980 kB/s | 32 kB 00:00 2024-09-12T15:53:03.259 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 843 kB/s | 14 kB 00:00 2024-09-12T15:53:03.285 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.7 MB/s | 220 kB 00:00 2024-09-12T15:53:03.310 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 355 kB/s | 279 kB 00:00 2024-09-12T15:53:03.368 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 12 MB/s | 1.0 MB 00:00 2024-09-12T15:53:03.393 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-09-12T15:53:03.411 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 5.2 MB/s | 90 kB 00:00 2024-09-12T15:53:03.494 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 86 kB/s | 54 kB 00:00 2024-09-12T15:53:03.498 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:03.498 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-09-12T15:53:03.571 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:03.588 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:03.589 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:03.687 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:03.688 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:04.114 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-12T15:53:04.139 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T15:53:04.227 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:04.410 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-12T15:53:04.481 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-12T15:53:04.537 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-12T15:53:04.572 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-12T15:53:04.633 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-12T15:53:04.739 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-12T15:53:04.849 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-12T15:53:04.900 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-12T15:53:05.670 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-12T15:53:05.721 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-12T15:53:05.754 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-12T15:53:05.804 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-12T15:53:05.832 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T15:53:06.536 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T15:53:06.536 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 1/13 2024-09-12T15:53:06.536 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-12T15:53:06.537 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-12T15:53:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 12/12 2024-09-12T15:53:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0 1/12 2024-09-12T15:53:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-12T15:53:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-12T15:53:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-12T15:53:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-12T15:53:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-12T15:53:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-12T15:53:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-12T15:53:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-12T15:53:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-12T15:53:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-12T15:53:06.885 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-12T15:53:06.885 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:06.885 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:06.885 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:06.885 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-12T15:53:06.886 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:06.887 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:07.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2024-09-12T15:53:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-12T15:53:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:07.216 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-12T15:53:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-12T15:53:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:07.217 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:07.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-rook 2024-09-12T15:53:07.736 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:28 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:07.877 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 154 k 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-12T15:53:07.878 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:07.879 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:07.879 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:07.879 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-09-12T15:53:07.879 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:07.879 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 7.0 M 2024-09-12T15:53:07.879 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2024-09-12T15:53:07.879 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:08.129 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-12T15:53:08.155 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4883.gd61b0886.e 694 kB/s | 154 kB 00:00 2024-09-12T15:53:08.213 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-12T15:53:08.238 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 691 kB/s | 58 kB 00:00 2024-09-12T15:53:08.255 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:25 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:08.264 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-12T15:53:08.394 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 49 k 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-12T15:53:08.396 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:08.397 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:08.398 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2024-09-12T15:53:08.398 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:08.398 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.2 M 2024-09-12T15:53:08.398 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-09-12T15:53:08.398 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:08.807 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 211 kB/s | 19 kB 00:00 2024-09-12T15:53:08.832 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 221 kB/s | 26 kB 00:00 2024-09-12T15:53:08.890 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): ceph-mgr-rook-19.3.0-4883.gd61b0886.el9 282 kB/s | 49 kB 00:00 2024-09-12T15:53:08.941 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-12T15:53:08.966 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-12T15:53:08.999 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-09-12T15:53:09.033 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-12T15:53:09.058 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-12T15:53:09.084 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-12T15:53:09.109 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-12T15:53:09.134 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-09-12T15:53:09.160 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-12T15:53:09.202 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-12T15:53:09.203 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:09.204 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-09-12T15:53:09.238 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-09-12T15:53:09.242 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:09.242 INFO:teuthology.orchestra.run.smithi019.stdout:Total 5.1 MB/s | 7.0 MB 00:01 2024-09-12T15:53:09.271 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:09.287 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:09.287 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:09.317 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:09.330 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:09.330 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:09.383 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:09.385 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:09.436 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:09.437 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:09.760 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:09.939 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-12T15:53:09.992 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-12T15:53:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:10.183 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-12T15:53:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-12T15:53:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-12T15:53:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-12T15:53:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-12T15:53:10.486 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-12T15:53:10.557 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-12T15:53:10.596 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-12T15:53:10.638 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-12T15:53:10.657 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-12T15:53:10.723 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-12T15:53:10.753 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T15:53:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-12T15:53:11.462 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-12T15:53:11.496 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-12T15:53:11.518 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T15:53:11.519 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/6 2024-09-12T15:53:11.519 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-12T15:53:11.519 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-12T15:53:11.519 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-12T15:53:11.519 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-12T15:53:11.545 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-12T15:53:11.573 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T15:53:11.817 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-12T15:53:11.817 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:11.820 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:11.821 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:11.821 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-12T15:53:11.821 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-12T15:53:11.821 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-12T15:53:11.821 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-12T15:53:11.822 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-12T15:53:11.822 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:11.822 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:12.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2024-09-12T15:53:12.304 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 13/13 2024-09-12T15:53:12.304 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 1/13 2024-09-12T15:53:12.304 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-12T15:53:12.304 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-12T15:53:12.304 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-12T15:53:12.304 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-12T15:53:12.304 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-12T15:53:12.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-12T15:53:12.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-12T15:53:12.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-12T15:53:12.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-12T15:53:12.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-12T15:53:12.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-12T15:53:12.639 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-12T15:53:12.639 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-12T15:53:12.640 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-12T15:53:12.641 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-12T15:53:12.641 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-12T15:53:12.641 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:12.641 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:12.665 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:33 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:12.805 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:12.805 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:12.805 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:53:12.805 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:12.805 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:12.805 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 900 k 2024-09-12T15:53:12.805 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 980 k 2024-09-12T15:53:12.806 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.9 M 2024-09-12T15:53:12.807 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:12.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-cephadm 2024-09-12T15:53:12.974 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 590 kB/s | 80 kB 00:00 2024-09-12T15:53:13.107 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-19.3.0-4883.gd61b0886.el9.x86_ 3.3 MB/s | 900 kB 00:00 2024-09-12T15:53:13.108 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:13.108 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.2 MB/s | 980 kB 00:00 2024-09-12T15:53:13.120 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:13.144 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:13.144 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:13.230 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:13.230 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:13.425 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:13.509 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:30 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:13.556 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-12T15:53:13.634 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T15:53:13.650 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:13.651 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:53:13.651 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:13.651 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:13.651 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 154 k 2024-09-12T15:53:13.651 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-09-12T15:53:13.652 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:13.653 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 7.0 M 2024-09-12T15:53:13.653 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 33 M 2024-09-12T15:53:13.653 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:14.146 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4883.gd61b0886.e 710 kB/s | 154 kB 00:00 2024-09-12T15:53:14.197 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-12T15:53:14.222 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-12T15:53:14.248 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-12T15:53:14.303 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T15:53:14.303 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/2 2024-09-12T15:53:14.564 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 391 kB/s | 248 kB 00:00 2024-09-12T15:53:14.592 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-12T15:53:14.592 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:14.593 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:14.593 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-12T15:53:14.593 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:14.593 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:14.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2024-09-12T15:53:15.474 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:36 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:15.613 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:15.614 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:15.614 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-09-12T15:53:15.614 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 285 k 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 285 k 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.3 M 2024-09-12T15:53:15.615 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:15.820 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-volume-19.3.0-4883.gd61b0886.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-12T15:53:15.821 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:15.821 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.4 MB/s | 285 kB 00:00 2024-09-12T15:53:15.821 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:15.830 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:15.830 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:15.859 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:15.859 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:16.089 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:16.189 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:53:16.206 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:53:16.206 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:53:16.206 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-12T15:53:16.206 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:16.781 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.1 MB/s | 6.0 MB 00:02 2024-09-12T15:53:16.783 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:16.783 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.2 MB/s | 7.0 MB 00:03 2024-09-12T15:53:16.860 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:16.873 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:16.873 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:16.979 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:16.980 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:17.232 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:53:17.232 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:17.232 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:17.233 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:17.233 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:17.233 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:17.530 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2024-09-12T15:53:17.621 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-12T15:53:17.668 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-12T15:53:18.129 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:39 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:18.227 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-12T15:53:18.269 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:18.269 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:18.269 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-12T15:53:18.269 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:18.269 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:18.270 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 127 k 2024-09-12T15:53:18.270 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:18.270 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:18.270 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:18.270 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:53:18.270 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:18.271 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 127 k 2024-09-12T15:53:18.271 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 456 k 2024-09-12T15:53:18.271 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:18.315 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-12T15:53:18.390 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-12T15:53:18.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T15:53:18.483 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-19.3.0-4883.gd61b0886.el9.x86_64 596 kB/s | 127 kB 00:00 2024-09-12T15:53:18.484 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:18.484 INFO:teuthology.orchestra.run.smithi019.stdout:Total 592 kB/s | 127 kB 00:00 2024-09-12T15:53:18.484 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:18.490 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:18.490 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:18.559 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:18.559 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:18.724 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:18.834 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:19.196 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 6/6 2024-09-12T15:53:19.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 1/6 2024-09-12T15:53:19.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-12T15:53:19.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-12T15:53:19.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-12T15:53:19.197 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-12T15:53:19.242 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-12T15:53:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:19.480 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-12T15:53:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-12T15:53:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-12T15:53:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-12T15:53:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-12T15:53:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:19.481 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:19.506 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:19.506 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:19.506 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:19.506 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:19.506 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:19.506 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:19.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2024-09-12T15:53:19.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse 2024-09-12T15:53:20.331 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:37 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:20.334 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:41 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:20.432 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:20.469 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:20.469 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 900 k 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-09-12T15:53:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:20.471 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 980 k 2024-09-12T15:53:20.471 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.9 M 2024-09-12T15:53:20.471 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:20.477 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:20.479 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:20.479 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:20.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2024-09-12T15:53:20.683 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 833 kB/s | 80 kB 00:00 2024-09-12T15:53:20.867 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-fuse-19.3.0-4883.gd61b0886.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-12T15:53:20.867 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:20.868 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.4 MB/s | 980 kB 00:00 2024-09-12T15:53:20.880 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:20.903 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:20.903 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:20.989 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:20.989 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:21.152 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:42 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:21.292 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:21.292 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 32 k 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:53:21.293 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:21.294 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 32 k 2024-09-12T15:53:21.294 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 142 k 2024-09-12T15:53:21.294 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-12T15:53:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T15:53:21.446 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-19.3.0-4883.gd61b0886.el9.x86_6 209 kB/s | 32 kB 00:00 2024-09-12T15:53:21.447 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:21.447 INFO:teuthology.orchestra.run.smithi019.stdout:Total 206 kB/s | 32 kB 00:00 2024-09-12T15:53:21.447 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:21.451 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:21.451 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:21.472 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:21.472 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:21.566 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:21.688 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:22.064 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:22.097 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2/2 2024-09-12T15:53:22.098 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/2 2024-09-12T15:53:22.306 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:22.306 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:22.306 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:22.306 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:22.306 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:22.307 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:22.396 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-12T15:53:22.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:22.397 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:22.397 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-12T15:53:22.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:22.397 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:22.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2024-09-12T15:53:22.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-volume 2024-09-12T15:53:23.066 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:44 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:23.159 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:23.206 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:23.207 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:23.207 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:23.264 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:40 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:23.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.3.0-4883.gd61b0886.el9 ceph-noarch 285 k 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:53:23.404 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:23.405 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 285 k 2024-09-12T15:53:23.405 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.3 M 2024-09-12T15:53:23.405 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:23.639 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-volume-19.3.0-4883.gd61b0886.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-12T15:53:23.640 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:23.640 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-12T15:53:23.640 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:23.649 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:23.649 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:23.684 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:23.685 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:23.860 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:44 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:23.920 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:23.953 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:23.991 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:53:23.999 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:24.001 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:24.001 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:24.007 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:53:24.007 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:53:24.008 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-12T15:53:24.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:24.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2024-09-12T15:53:24.654 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:45 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:24.746 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:24.793 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:24.794 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:24.794 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:24.855 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2024-09-12T15:53:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 1/1 2024-09-12T15:53:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:25.057 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.3.0-4883.gd61b0886.el9.noarch 2024-09-12T15:53:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:25.057 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:25.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel 2024-09-12T15:53:25.453 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:46 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:25.548 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:25.594 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:25.595 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:25.595 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:25.656 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2024-09-12T15:53:25.935 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:42 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:26.077 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:26.077 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 127 k 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 127 k 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 456 k 2024-09-12T15:53:26.078 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:26.247 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:47 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:26.261 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-19.3.0-4883.gd61b0886.el9.x86_64 694 kB/s | 127 kB 00:00 2024-09-12T15:53:26.262 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:26.262 INFO:teuthology.orchestra.run.smithi016.stdout:Total 688 kB/s | 127 kB 00:00 2024-09-12T15:53:26.262 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:26.268 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:26.268 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:26.337 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:26.337 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:26.341 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:26.388 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:26.389 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:26.389 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:26.453 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2024-09-12T15:53:26.496 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:26.613 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:27.022 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:27.051 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:48 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:27.144 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:27.190 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:27.191 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:27.191 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:27.264 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2024-09-12T15:53:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:27.298 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:27.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:27.298 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:27.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2 2024-09-12T15:53:27.856 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:48 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:27.996 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:27.997 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:27.997 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-09-12T15:53:27.997 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:27.997 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 85 k 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 85 k 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 231 k 2024-09-12T15:53:27.998 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:28.191 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-19.3.0-4883.gd61b0886.el9.x86_64.rpm 442 kB/s | 85 kB 00:00 2024-09-12T15:53:28.191 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:28.192 INFO:teuthology.orchestra.run.smithi019.stdout:Total 439 kB/s | 85 kB 00:00 2024-09-12T15:53:28.192 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:28.192 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:45 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:28.199 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:28.199 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:28.266 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:28.266 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:28.286 INFO:teuthology.orchestra.run.smithi016.stdout:Package libcephfs2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:28.333 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:28.334 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:28.334 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:28.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs-devel 2024-09-12T15:53:28.441 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:28.542 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:28.920 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:28.984 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:45 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:29.124 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:29.124 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:29.124 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-12T15:53:29.124 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:29.124 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:29.125 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 32 k 2024-09-12T15:53:29.125 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:29.125 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:29.125 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:29.125 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:53:29.125 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:29.126 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 32 k 2024-09-12T15:53:29.126 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 142 k 2024-09-12T15:53:29.126 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:29.199 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:29.199 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:29.200 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:29.200 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:29.200 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:29.200 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:29.266 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs-devel-19.3.0-4883.gd61b0886.el9.x86_6 227 kB/s | 32 kB 00:00 2024-09-12T15:53:29.266 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:29.266 INFO:teuthology.orchestra.run.smithi016.stdout:Total 225 kB/s | 32 kB 00:00 2024-09-12T15:53:29.267 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:29.270 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:29.270 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:29.297 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:29.297 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:29.465 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-09-12T15:53:29.491 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:30.062 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:30.136 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:30.136 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:30.212 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:30.213 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:30.213 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-09-12T15:53:30.213 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.2 M 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 14 M 2024-09-12T15:53:30.214 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:30.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2 2024-09-12T15:53:30.528 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-19.3.0-4883.gd61b0886.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-12T15:53:30.528 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:30.529 INFO:teuthology.orchestra.run.smithi019.stdout:Total 10 MB/s | 3.2 MB 00:00 2024-09-12T15:53:30.529 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:30.539 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:30.539 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:30.626 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:30.626 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:30.898 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:47 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:30.990 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:31.036 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:31.040 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:31.040 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:31.102 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:31.104 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1 2024-09-12T15:53:31.162 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:31.189 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:31.189 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:53:31.189 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-12T15:53:31.189 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T15:53:31.190 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T15:53:31.190 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:31.689 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:48 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:31.783 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:31.829 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:31.830 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:31.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados 2024-09-12T15:53:32.348 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:32.348 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:32.348 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:32.349 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:32.349 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:32.349 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:32.494 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:49 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:32.556 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2024-09-12T15:53:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rados-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:32.633 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:32.634 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:32.634 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:32.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw 2024-09-12T15:53:33.150 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:54 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:33.289 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:33.289 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:33.289 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-09-12T15:53:33.289 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:33.289 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-09-12T15:53:33.289 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 171 k 2024-09-12T15:53:33.289 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:33.290 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-09-12T15:53:33.290 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-09-12T15:53:33.290 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-09-12T15:53:33.290 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:33.290 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 171 k 2024-09-12T15:53:33.290 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 490 k 2024-09-12T15:53:33.290 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-09-12T15:53:33.292 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:50 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:33.388 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rgw-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:33.435 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:33.436 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:33.436 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:33.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs 2024-09-12T15:53:33.556 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-19.3.0-4883.gd61b0886.el9.x86_64.rpm 645 kB/s | 171 kB 00:00 2024-09-12T15:53:33.556 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:33.557 INFO:teuthology.orchestra.run.smithi019.stdout:Total 641 kB/s | 171 kB 00:00 2024-09-12T15:53:33.557 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-09-12T15:53:33.565 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-09-12T15:53:33.565 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-09-12T15:53:33.633 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-09-12T15:53:33.633 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-09-12T15:53:33.792 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-09-12T15:53:33.907 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:34.087 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:34.181 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-cephfs-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:34.226 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:34.227 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:34.228 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:34.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd 2024-09-12T15:53:34.317 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:34.581 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:34.581 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:34.581 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-09-12T15:53:34.581 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:34.581 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:34.581 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:34.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-09-12T15:53:34.894 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:51 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:34.989 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rbd-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:35.035 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:35.036 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:35.036 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:35.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse 2024-09-12T15:53:35.404 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:56 ago on Thu 12 Sep 2024 03:51:39 PM UTC. 2024-09-12T15:53:35.497 INFO:teuthology.orchestra.run.smithi019.stdout:Package rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:35.544 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-09-12T15:53:35.545 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-09-12T15:53:35.545 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-09-12T15:53:35.616 DEBUG:teuthology.parallel:result is None 2024-09-12T15:53:35.710 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:52 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:35.852 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:35.853 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:35.853 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-12T15:53:35.853 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 85 k 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 85 k 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 231 k 2024-09-12T15:53:35.854 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:36.053 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-19.3.0-4883.gd61b0886.el9.x86_64.rpm 430 kB/s | 85 kB 00:00 2024-09-12T15:53:36.053 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:36.053 INFO:teuthology.orchestra.run.smithi016.stdout:Total 426 kB/s | 85 kB 00:00 2024-09-12T15:53:36.054 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:36.061 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:36.061 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:36.127 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:36.128 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:36.278 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:36.386 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:36.788 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:37.053 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:37.053 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:37.053 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:37.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2024-09-12T15:53:37.948 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:54 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:38.090 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:38.091 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-09-12T15:53:38.091 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:38.091 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 3.2 M 2024-09-12T15:53:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:38.092 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:38.092 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:38.092 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:53:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:38.092 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.2 M 2024-09-12T15:53:38.092 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 14 M 2024-09-12T15:53:38.092 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:38.407 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-19.3.0-4883.gd61b0886.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-12T15:53:38.407 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:38.408 INFO:teuthology.orchestra.run.smithi016.stdout:Total 10 MB/s | 3.2 MB 00:00 2024-09-12T15:53:38.408 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:38.418 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:38.418 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:38.504 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:38.504 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:38.968 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:39.033 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:39.060 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:39.060 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-12T15:53:39.060 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-12T15:53:39.060 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T15:53:39.060 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-12T15:53:39.061 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:40.302 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:40.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:40.302 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:40.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd 2024-09-12T15:53:41.116 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:58 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:41.256 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.3.0-4883.gd61b0886.el9 ceph 171 k 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-09-12T15:53:41.257 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-09-12T15:53:41.258 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-09-12T15:53:41.258 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:41.258 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 171 k 2024-09-12T15:53:41.258 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 490 k 2024-09-12T15:53:41.258 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-09-12T15:53:41.469 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-19.3.0-4883.gd61b0886.el9.x86_64.rpm 810 kB/s | 171 kB 00:00 2024-09-12T15:53:41.470 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-09-12T15:53:41.470 INFO:teuthology.orchestra.run.smithi016.stdout:Total 804 kB/s | 171 kB 00:00 2024-09-12T15:53:41.470 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-09-12T15:53:41.479 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-09-12T15:53:41.479 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-09-12T15:53:41.546 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-09-12T15:53:41.547 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-09-12T15:53:41.713 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-09-12T15:53:41.828 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:42.257 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:42.539 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 1/1 2024-09-12T15:53:42.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:42.539 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-09-12T15:53:42.539 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.3.0-4883.gd61b0886.el9.x86_64 2024-09-12T15:53:42.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:42.539 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:42.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2024-09-12T15:53:43.373 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:02:00 ago on Thu 12 Sep 2024 03:51:43 PM UTC. 2024-09-12T15:53:43.466 INFO:teuthology.orchestra.run.smithi016.stdout:Package rbd-mirror-2:19.3.0-4883.gd61b0886.el9.x86_64 is already installed. 2024-09-12T15:53:43.512 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-09-12T15:53:43.513 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-09-12T15:53:43.513 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-09-12T15:53:43.582 DEBUG:teuthology.parallel:result is None 2024-09-12T15:53:43.582 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T15:53:43.724 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-12T15:53:43.746 INFO:teuthology.orchestra.run.smithi016.stdout:19.3.0-4883.gd61b0886.el9 2024-09-12T15:53:43.747 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883.gd61b0886.el9 2024-09-12T15:53:43.747 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883.gd61b0886 is installed. 2024-09-12T15:53:43.749 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T15:53:43.884 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-12T15:53:43.911 INFO:teuthology.orchestra.run.smithi019.stdout:19.3.0-4883.gd61b0886.el9 2024-09-12T15:53:43.911 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883.gd61b0886.el9 2024-09-12T15:53:43.911 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883.gd61b0886 is installed. 2024-09-12T15:53:43.913 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-12T15:53:43.913 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:43.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T15:53:43.940 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:43.940 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-12T15:53:43.981 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-12T15:53:43.981 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:43.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T15:53:44.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T15:53:44.075 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:44.076 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-09-12T15:53:44.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-12T15:53:44.176 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-12T15:53:44.177 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:44.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T15:53:44.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T15:53:44.271 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:44.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-12T15:53:44.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-12T15:53:44.372 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-12T15:53:44.372 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:44.372 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T15:53:44.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T15:53:44.468 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:44.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-09-12T15:53:44.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-12T15:53:44.570 INFO:teuthology.run_tasks:Running task ceph... 2024-09-12T15:53:44.678 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-12T15:53:44.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2024-09-12T15:53:44.682 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2024-09-12T15:53:44.712 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-12T15:53:44.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-12T15:53:44.751 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-12T15:53:44.781 INFO:tasks.ceph:Creating extra log directories... 2024-09-12T15:53:44.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-12T15:53:44.820 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-12T15:53:44.864 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-12T15:53:44.865 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-12T15:53:44.865 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7902659', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1680, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'sleep_before_teardown': 0, 'subset': '42/100', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOGlv8KdgRy99aYpxFzc9OxbFOhe+Psd2fGj+dWHhm0eXiEVM74bK2aUhC+j8nwkTndUeu+LbLlyO70xx6cAFLA=', 'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnPQM/fgXjYviHXSJUTvWuMdxI2P0i0UhacxnQunjFI8ccRd23NudHOCUn715FX1le15xAEncHuHd6ajf78PJI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-12_15:21:08', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-12T15:53:44.865 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-12T15:53:44.888 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-12T15:53:44.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T15:53:44.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T15:53:44.978 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:44.979 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T15:53:45.028 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T15:53:45.029 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 15:53:39.340623218 +0000 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 15:51:01.366384551 +0000 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 15:51:01.366384551 +0000 2024-09-12T15:53:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 15:51:01.366384551 +0000 2024-09-12T15:53:45.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T15:53:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T15:53:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T15:53:45.153 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000104236 s, 4.9 MB/s 2024-09-12T15:53:45.154 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T15:53:45.210 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 15:53:39.340623218 +0000 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 15:51:01.687379700 +0000 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 15:51:01.687379700 +0000 2024-09-12T15:53:45.268 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 15:51:01.687379700 +0000 2024-09-12T15:53:45.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T15:53:45.333 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T15:53:45.333 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T15:53:45.334 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000101426 s, 5.0 MB/s 2024-09-12T15:53:45.335 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T15:53:45.392 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-09-12T15:53:45.448 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T15:53:45.448 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:53:45.449 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-12T15:53:45.449 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:53:45.449 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:53:45.449 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 15:53:39.341623203 +0000 2024-09-12T15:53:45.449 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 15:51:02.016374729 +0000 2024-09-12T15:53:45.449 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 15:51:02.016374729 +0000 2024-09-12T15:53:45.449 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 15:51:02.016374729 +0000 2024-09-12T15:53:45.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T15:53:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T15:53:45.513 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T15:53:45.514 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000103681 s, 4.9 MB/s 2024-09-12T15:53:45.515 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T15:53:45.572 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-09-12T15:53:45.628 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T15:53:45.628 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:53:45.628 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-12T15:53:45.628 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:53:45.629 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:53:45.629 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-12 15:53:39.341623203 +0000 2024-09-12T15:53:45.629 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-12 15:51:02.339369848 +0000 2024-09-12T15:53:45.629 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-12 15:51:02.339369848 +0000 2024-09-12T15:53:45.629 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-09-12 15:51:02.339369848 +0000 2024-09-12T15:53:45.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T15:53:45.692 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-12T15:53:45.692 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-12T15:53:45.692 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000105039 s, 4.9 MB/s 2024-09-12T15:53:45.694 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T15:53:45.751 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:53:45.751 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T15:53:45.751 INFO:tasks.ceph:Generating config... 2024-09-12T15:53:45.752 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-12T15:53:45.752 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-12T15:53:45.752 INFO:tasks.ceph:[client] log to stderr = False 2024-09-12T15:53:45.752 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:45.753 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:45.753 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-12T15:53:45.753 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-12T15:53:45.753 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-12T15:53:45.753 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-12T15:53:45.754 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-12T15:53:45.754 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-12T15:53:45.754 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-12T15:53:45.754 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-12T15:53:45.754 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-12T15:53:45.754 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-12T15:53:45.754 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-12T15:53:45.754 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-12T15:53:45.754 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-12T15:53:45.754 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-12T15:53:45.754 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-12T15:53:45.755 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-12T15:53:45.755 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-12T15:53:45.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-12T15:53:45.840 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/cluster1.keyring 2024-09-12T15:53:45.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-12T15:53:45.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-12T15:53:45.913 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.16')] 2024-09-12T15:53:45.913 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-12T15:53:45.914 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:45.914 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T15:53:45.969 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid b23de1a0-bb6d-4bd1-97d5-57131ff67d17 2024-09-12T15:53:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-09-12T15:53:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2024-09-12T15:53:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:fsid b23de1a0-bb6d-4bd1-97d5-57131ff67d17 2024-09-12T15:53:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2024-09-12T15:53:46.051195+0000 2024-09-12T15:53:46.051 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-12T15:53:46.051195+0000 2024-09-12T15:53:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 17 (quincy) 2024-09-12T15:53:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2024-09-12T15:53:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-12T15:53:46.052 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-12T15:53:46.053 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T15:53:46.109 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID b23de1a0-bb6d-4bd1-97d5-57131ff67d17... 2024-09-12T15:53:46.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-12T15:53:46.151 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-09-12T15:53:46.194 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-09-12T15:53:46.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.196 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = 172.21.15.16 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: mon client directed command retry = 5 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on pool no app = False 2024-09-12T15:53:46.197 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject socket failures = 5000 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = b23de1a0-bb6d-4bd1-97d5-57131ff67d17 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.198 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-09-12T15:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-09-12T15:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: osd memory target = 1610612736 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-09-12T15:53:46.201 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-12T15:53:46.202 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T15:53:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T15:53:46.204 INFO:teuthology.orchestra.run.smithi016.stdout: debug rbd = 20 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: debug rbd_mirror = 15 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: log to stderr = False 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: rbd default features = 125 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: rbd mirror image policy type = none 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.0] 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.1] 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.2] 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.205 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.206 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.3] 2024-09-12T15:53:46.206 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.206 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.206 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-09-12T15:53:46.206 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-09-12T15:53:46.206 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.207 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-09-12T15:53:46.208 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-09-12T15:53:46.209 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = 172.21.15.16 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: mon client directed command retry = 5 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on pool no app = False 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: ms inject socket failures = 5000 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = b23de1a0-bb6d-4bd1-97d5-57131ff67d17 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-09-12T15:53:46.210 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.211 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: bdev async discard = True 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: bdev enable discard = True 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2024-09-12T15:53:46.212 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 1/20 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 1/20 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 4/10 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: osd memory target = 1610612736 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.213 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-09-12T15:53:46.214 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.215 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T15:53:46.216 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd = 20 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd_mirror = 15 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: log to stderr = False 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: rbd default features = 125 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: rbd mirror image policy type = none 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.0] 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.1] 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.217 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.2] 2024-09-12T15:53:46.218 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.218 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.218 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.3] 2024-09-12T15:53:46.218 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:53:46.218 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:53:46.219 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-12T15:53:46.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-12T15:53:46.315 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-12T15:53:46.316 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:46.316 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-12T15:53:46.332 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:46.332 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-12T15:53:46.387 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T15:53:46.387 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:46.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-12T15:53:46.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-12T15:53:46.466 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:46.466 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:46.522 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T15:53:46.522 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:46.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-12T15:53:46.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-12T15:53:46.565 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:46.565 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:46.624 INFO:tasks.ceph:Setting up mon nodes... 2024-09-12T15:53:46.624 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-12T15:53:46.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-12T15:53:46.678 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-12T15:53:46.680 INFO:tasks.ceph:Setting up mds nodes... 2024-09-12T15:53:46.681 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-12T15:53:46.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-12T15:53:46.723 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-12T15:53:46.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-12T15:53:46.795 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-12T15:53:46.809 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-12T15:53:46.854 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-12T15:53:46.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-12T15:53:46.909 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-12T15:53:46.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-12T15:53:47.006 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-12T15:53:47.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-12T15:53:47.102 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-12T15:53:47.115 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-12T15:53:47.197 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-12T15:53:47.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-12T15:53:47.293 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-12T15:53:47.306 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-12T15:53:47.388 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-12T15:53:47.401 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-12T15:53:47.401 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T15:53:47.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-12T15:53:47.431 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:53:47.431 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-12T15:53:47.431 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T15:53:47.431 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-12T15:53:47.500 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T15:53:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T15:53:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T15:53:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T15:53:47.500 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T15:53:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-12T15:53:47.500 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T15:53:47.501 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T15:53:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T15:53:47.501 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T15:53:47.532 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-12T15:53:47.535 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-12T15:53:47.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-12T15:53:47.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-12T15:53:47.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-12T15:53:47.745 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:53:47.745 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-12T15:53:47.745 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T15:53:47.745 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T15:53:47.813 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T15:53:47.845 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-12T15:53:47.847 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-12T15:53:47.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-12T15:53:47.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-12T15:53:47.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-12T15:53:48.056 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:53:48.057 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-12T15:53:48.057 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T15:53:48.057 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T15:53:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T15:53:48.124 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T15:53:48.155 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-09-12T15:53:48.157 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-12T15:53:48.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-12T15:53:48.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-12T15:53:48.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:48.395+0000 7fd96cc18640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-12T15:53:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:48.395+0000 7fd96cc18640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-12T15:53:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:48.395+0000 7fd96cc18640 -1 bdev(0x55872071b800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-12T15:53:48.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:48.396+0000 7fd96cc18640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-12T15:53:51.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-12T15:53:51.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:51.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:51.676+0000 7f9a0f224640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-12T15:53:51.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:51.676+0000 7f9a0f224640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-12T15:53:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:51.676+0000 7f9a0f224640 -1 bdev(0x5594fd4a7800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-12T15:53:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:51.676+0000 7f9a0f224640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-12T15:53:54.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-12T15:53:54.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:54.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:54.711+0000 7f820adae640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-12T15:53:54.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:54.711+0000 7f820adae640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-12T15:53:54.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:54.711+0000 7f820adae640 -1 bdev(0x563663965800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-12T15:53:54.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-12T15:53:54.711+0000 7f820adae640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-12T15:53:57.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-12T15:53:57.659 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-12T15:53:57.659 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:57.659 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-12T15:53:57.728 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:57.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-12T15:53:57.796 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:57.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-12T15:53:57.863 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:57.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-12T15:53:57.931 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:57.932 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-12T15:53:57.989 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:57.989 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-12T15:53:58.009 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:58.009 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-12T15:53:58.071 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:58.072 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-12T15:53:58.130 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:58.130 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-12T15:53:58.189 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:58.190 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-12T15:53:58.249 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:58.249 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-12T15:53:58.307 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:58.307 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-12T15:53:58.365 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:53:58.365 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-12T15:53:58.422 INFO:tasks.ceph:Adding keys to all mons... 2024-09-12T15:53:58.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCKDuNm5YFqKBAAi6HcMX5QTp2rdOEPIHHSXA== 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCMDuNmLj2nFxAA1txVXZAMvJYhSyk37I8wXA== 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCPDuNmlcBdKBAAZqkIkHgqlonWXyZ+7s0rSA== 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2024-09-12T15:53:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCSDuNmQ+N4KhAAZ9m0U2v0AxbtGx0WhrQ9zw== 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCKDuNmpIYiKxAAwtVtuHcPtJyXhjq9P8XZ3g== 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror] 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCKDuNmMCJwLxAAy//QuVBPt6cMhxoXt8TDEw== 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.0] 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCKDuNmmy/oMhAAtIh13R2FYm/iIOhWdMqX8A== 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.1] 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCKDuNmSkU+NhAAVaYrusdKgeLRjijV+7SpzA== 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.2] 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCLDuNmBxthABAAo7+yHGhe7Q7jpPvQqbgwag== 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.3] 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCLDuNmRmgaBhAAE9Rm6nGFw62PMkzau6B2lA== 2024-09-12T15:53:58.453 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.4] 2024-09-12T15:53:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCLDuNmQMXICxAApSFhv7NNNpBbjAmHdxHzGQ== 2024-09-12T15:53:58.454 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.5] 2024-09-12T15:53:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCLDuNmrQB4ERAAr5bkXMIDlJtv6mXgDRVdsw== 2024-09-12T15:53:58.454 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.6] 2024-09-12T15:53:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCLDuNms6YkFxAA5Z43iaF49l6piMJ2s2cRow== 2024-09-12T15:53:58.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-12T15:53:58.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T15:53:58.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T15:53:58.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T15:53:58.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:58.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:58.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:58.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:58.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:58.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:59.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:59.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:59.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:53:59.215 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-12T15:53:59.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-12T15:53:59.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-12T15:53:59.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-12T15:53:59.363 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-12T15:53:59.419 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-12T15:53:59.420 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-12T15:53:59.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-12T15:53:59.462 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-12T15:53:59.463 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-12T15:53:59.463 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-12T15:53:59.463 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-12T15:53:59.465 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-12T15:53:59.466 DEBUG:tasks.ceph:set 0 configs 2024-09-12T15:53:59.466 INFO:tasks.ceph:Setting crush tunables to default 2024-09-12T15:53:59.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-12T15:53:59.643 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-09-12T15:53:59.654 INFO:tasks.ceph:check_enable_crimson: False 2024-09-12T15:53:59.654 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-12T15:53:59.655 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:59.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-12T15:53:59.680 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:59.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-12T15:53:59.746 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:53:59.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-12T15:53:59.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster cluster1 osd new b2070859-ce30-4b8b-bc50-4d7852c29e68 0 2024-09-12T15:54:00.037 INFO:teuthology.orchestra.run.smithi016.stdout:0 2024-09-12T15:54:00.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster cluster1 osd new 6a5979a6-605b-4a99-a7e6-d3950b5e8e86 1 2024-09-12T15:54:00.229 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-09-12T15:54:00.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster cluster1 osd new 4cde2604-d7a6-4c26-8e59-21e0f79c0fa9 2 2024-09-12T15:54:00.423 INFO:teuthology.orchestra.run.smithi016.stdout:2 2024-09-12T15:54:00.434 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-12T15:54:00.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-12T15:54:00.437 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-12T15:54:00.437 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-12T15:54:00.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-12T15:54:00.439 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-12T15:54:00.440 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-12T15:54:00.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-12T15:54:00.447 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-12T15:54:00.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T15:54:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:00.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"fsid":"b23de1a0-bb6d-4bd1-97d5-57131ff67d17","created":"2024-09-12T15:53:59.551884+0000","modified":"2024-09-12T15:54:00.421731+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T15:54:00.421731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b2070859-ce30-4b8b-bc50-4d7852c29e68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6a5979a6-605b-4a99-a7e6-d3950b5e8e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4cde2604-d7a6-4c26-8e59-21e0f79c0fa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:00.669 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-12T15:54:00.670 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-12T15:54:00.972 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-12T15:54:01.137 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-12T15:54:01.137 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":5,"fsid":"b23de1a0-bb6d-4bd1-97d5-57131ff67d17","created":"2024-09-12T15:53:59.551884+0000","modified":"2024-09-12T15:54:00.421731+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T15:54:00.421731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b2070859-ce30-4b8b-bc50-4d7852c29e68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6a5979a6-605b-4a99-a7e6-d3950b5e8e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4cde2604-d7a6-4c26-8e59-21e0f79c0fa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:01.147 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T15:54:01.400 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T15:54:01.398+0000 7fae5bc2c640 -1 Falling back to public interface 2024-09-12T15:54:01.654 INFO:tasks.ceph.cluster1.osd.2.smithi016.stderr:2024-09-12T15:54:01.652+0000 7fd5b5642640 -1 Falling back to public interface 2024-09-12T15:54:01.895 INFO:tasks.ceph.cluster1.osd.1.smithi016.stderr:2024-09-12T15:54:01.893+0000 7fb0b6c0f640 -1 Falling back to public interface 2024-09-12T15:54:05.337 INFO:tasks.ceph.cluster1.mgr.x.smithi016.stderr:2024-09-12T15:54:05.335+0000 7f84735bd640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-12T15:54:05.463 INFO:tasks.ceph.cluster1.osd.1.smithi016.stderr:2024-09-12T15:54:05.462+0000 7fb0b6c0f640 -1 osd.1 0 log_to_monitors true 2024-09-12T15:54:05.493 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T15:54:05.491+0000 7fae5bc2c640 -1 osd.0 0 log_to_monitors true 2024-09-12T15:54:05.752 INFO:tasks.ceph.cluster1.osd.2.smithi016.stderr:2024-09-12T15:54:05.751+0000 7fd5b5642640 -1 osd.2 0 log_to_monitors true 2024-09-12T15:54:07.450 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-12T15:54:07.892 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-12T15:54:07.892 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":7,"fsid":"b23de1a0-bb6d-4bd1-97d5-57131ff67d17","created":"2024-09-12T15:53:59.551884+0000","modified":"2024-09-12T15:54:07.131929+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T15:54:00.421731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b2070859-ce30-4b8b-bc50-4d7852c29e68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6a5979a6-605b-4a99-a7e6-d3950b5e8e86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4cde2604-d7a6-4c26-8e59-21e0f79c0fa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:07.910 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T15:54:08.472 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T15:54:08.471+0000 7fae583cf640 -1 osd.0 0 waiting for initial osdmap 2024-09-12T15:54:08.482 INFO:tasks.ceph.cluster1.osd.2.smithi016.stderr:2024-09-12T15:54:08.481+0000 7fd5b15d1640 -1 osd.2 0 waiting for initial osdmap 2024-09-12T15:54:08.484 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T15:54:08.483+0000 7fae4f9da640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T15:54:08.485 INFO:tasks.ceph.cluster1.osd.1.smithi016.stderr:2024-09-12T15:54:08.484+0000 7fb0b261c640 -1 osd.1 0 waiting for initial osdmap 2024-09-12T15:54:08.499 INFO:tasks.ceph.cluster1.osd.1.smithi016.stderr:2024-09-12T15:54:08.498+0000 7fb0aa43b640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T15:54:08.499 INFO:tasks.ceph.cluster1.osd.2.smithi016.stderr:2024-09-12T15:54:08.498+0000 7fd5a93f0640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T15:54:14.213 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-12T15:54:14.517 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-12T15:54:14.517 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":11,"fsid":"b23de1a0-bb6d-4bd1-97d5-57131ff67d17","created":"2024-09-12T15:53:59.551884+0000","modified":"2024-09-12T15:54:12.485659+0000","last_up_change":"2024-09-12T15:54:09.483002+0000","last_in_change":"2024-09-12T15:54:00.421731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T15:54:10.147273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2070859-ce30-4b8b-bc50-4d7852c29e68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6801","nonce":196569321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6803","nonce":196569321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6807","nonce":196569321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6805","nonce":196569321}]},"public_addr":"172.21.15.16:6801/196569321","cluster_addr":"172.21.15.16:6803/196569321","heartbeat_back_addr":"172.21.15.16:6807/196569321","heartbeat_front_addr":"172.21.15.16:6805/196569321","state":["exists","up"]},{"osd":1,"uuid":"6a5979a6-605b-4a99-a7e6-d3950b5e8e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6817","nonce":2469380610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6819","nonce":2469380610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6823","nonce":2469380610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6821","nonce":2469380610}]},"public_addr":"172.21.15.16:6817/2469380610","cluster_addr":"172.21.15.16:6819/2469380610","heartbeat_back_addr":"172.21.15.16:6823/2469380610","heartbeat_front_addr":"172.21.15.16:6821/2469380610","state":["exists","up"]},{"osd":2,"uuid":"4cde2604-d7a6-4c26-8e59-21e0f79c0fa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6809","nonce":791970945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6811","nonce":791970945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6815","nonce":791970945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6813","nonce":791970945}]},"public_addr":"172.21.15.16:6809/791970945","cluster_addr":"172.21.15.16:6811/791970945","heartbeat_back_addr":"172.21.15.16:6815/791970945","heartbeat_front_addr":"172.21.15.16:6813/791970945","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.498009+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.468066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.772776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:14.528 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-12T15:54:14.528 INFO:tasks.ceph:Creating RBD pool 2024-09-12T15:54:14.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-12T15:54:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'rbd' created 2024-09-12T15:54:15.511 DEBUG:teuthology.orchestra.run.smithi016:> rbd --cluster cluster1 pool init rbd 2024-09-12T15:54:18.514 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-12T15:54:18.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 config log 1 --format=json 2024-09-12T15:54:18.515 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-12T15:54:18.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:18.839 INFO:teuthology.orchestra.run.smithi016.stdout:[{"version":4,"timestamp":"2024-09-12T15:54:08.485589+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"4402.674142"}]}] 2024-09-12T15:54:18.839 INFO:tasks.ceph_manager:config epoch is 4 2024-09-12T15:54:18.839 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-12T15:54:18.839 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-12T15:54:18.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-12T15:54:19.186 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:19.196 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":773707199},{"type":"v1","addr":"172.21.15.16:6825","nonce":773707199}]},"active_addr":"172.21.15.16:6825/773707199","active_change":"2024-09-12T15:54:04.127895+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1223123928}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":998232184}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":603741308}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":705408399}]}]} 2024-09-12T15:54:19.199 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-12T15:54:19.199 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-12T15:54:19.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T15:54:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:19.497 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":15,"fsid":"b23de1a0-bb6d-4bd1-97d5-57131ff67d17","created":"2024-09-12T15:53:59.551884+0000","modified":"2024-09-12T15:54:18.499938+0000","last_up_change":"2024-09-12T15:54:09.483002+0000","last_in_change":"2024-09-12T15:54:00.421731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T15:54:10.147273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T15:54:14.828915+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2070859-ce30-4b8b-bc50-4d7852c29e68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6801","nonce":196569321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6803","nonce":196569321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6807","nonce":196569321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6805","nonce":196569321}]},"public_addr":"172.21.15.16:6801/196569321","cluster_addr":"172.21.15.16:6803/196569321","heartbeat_back_addr":"172.21.15.16:6807/196569321","heartbeat_front_addr":"172.21.15.16:6805/196569321","state":["exists","up"]},{"osd":1,"uuid":"6a5979a6-605b-4a99-a7e6-d3950b5e8e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6817","nonce":2469380610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6819","nonce":2469380610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6823","nonce":2469380610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6821","nonce":2469380610}]},"public_addr":"172.21.15.16:6817/2469380610","cluster_addr":"172.21.15.16:6819/2469380610","heartbeat_back_addr":"172.21.15.16:6823/2469380610","heartbeat_front_addr":"172.21.15.16:6821/2469380610","state":["exists","up"]},{"osd":2,"uuid":"4cde2604-d7a6-4c26-8e59-21e0f79c0fa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6809","nonce":791970945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6811","nonce":791970945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6815","nonce":791970945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6813","nonce":791970945}]},"public_addr":"172.21.15.16:6809/791970945","cluster_addr":"172.21.15.16:6811/791970945","heartbeat_back_addr":"172.21.15.16:6815/791970945","heartbeat_front_addr":"172.21.15.16:6813/791970945","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.498009+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.468066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.772776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:19.510 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-12T15:54:19.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T15:54:19.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:19.807 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":15,"fsid":"b23de1a0-bb6d-4bd1-97d5-57131ff67d17","created":"2024-09-12T15:53:59.551884+0000","modified":"2024-09-12T15:54:18.499938+0000","last_up_change":"2024-09-12T15:54:09.483002+0000","last_in_change":"2024-09-12T15:54:00.421731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T15:54:10.147273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T15:54:14.828915+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2070859-ce30-4b8b-bc50-4d7852c29e68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6801","nonce":196569321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6803","nonce":196569321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6807","nonce":196569321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6805","nonce":196569321}]},"public_addr":"172.21.15.16:6801/196569321","cluster_addr":"172.21.15.16:6803/196569321","heartbeat_back_addr":"172.21.15.16:6807/196569321","heartbeat_front_addr":"172.21.15.16:6805/196569321","state":["exists","up"]},{"osd":1,"uuid":"6a5979a6-605b-4a99-a7e6-d3950b5e8e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6817","nonce":2469380610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6819","nonce":2469380610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6823","nonce":2469380610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6821","nonce":2469380610}]},"public_addr":"172.21.15.16:6817/2469380610","cluster_addr":"172.21.15.16:6819/2469380610","heartbeat_back_addr":"172.21.15.16:6823/2469380610","heartbeat_front_addr":"172.21.15.16:6821/2469380610","state":["exists","up"]},{"osd":2,"uuid":"4cde2604-d7a6-4c26-8e59-21e0f79c0fa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6809","nonce":791970945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6811","nonce":791970945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6815","nonce":791970945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6813","nonce":791970945}]},"public_addr":"172.21.15.16:6809/791970945","cluster_addr":"172.21.15.16:6811/791970945","heartbeat_back_addr":"172.21.15.16:6815/791970945","heartbeat_front_addr":"172.21.15.16:6813/791970945","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.498009+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.468066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.772776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:19.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-12T15:54:19.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-12T15:54:19.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-12T15:54:19.962 INFO:teuthology.orchestra.run.smithi016.stdout:34359738372 2024-09-12T15:54:19.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-12T15:54:19.981 INFO:teuthology.orchestra.run.smithi016.stdout:34359738372 2024-09-12T15:54:19.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-12T15:54:19.982 INFO:teuthology.orchestra.run.smithi016.stdout:34359738371 2024-09-12T15:54:19.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-12T15:54:20.285 INFO:teuthology.orchestra.run.smithi016.stdout:34359738371 2024-09-12T15:54:20.298 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2024-09-12T15:54:20.328 INFO:teuthology.orchestra.run.smithi016.stdout:34359738370 2024-09-12T15:54:20.341 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2024-09-12T15:54:20.348 INFO:teuthology.orchestra.run.smithi016.stdout:34359738371 2024-09-12T15:54:20.361 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2024-09-12T15:54:21.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-12T15:54:21.342 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-12T15:54:21.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-12T15:54:21.623 INFO:teuthology.orchestra.run.smithi016.stdout:34359738372 2024-09-12T15:54:21.636 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2024-09-12T15:54:21.636 DEBUG:teuthology.parallel:result is None 2024-09-12T15:54:21.662 INFO:teuthology.orchestra.run.smithi016.stdout:34359738371 2024-09-12T15:54:21.674 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2024-09-12T15:54:21.674 DEBUG:teuthology.parallel:result is None 2024-09-12T15:54:21.705 INFO:teuthology.orchestra.run.smithi016.stdout:34359738372 2024-09-12T15:54:21.718 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2024-09-12T15:54:21.718 DEBUG:teuthology.parallel:result is None 2024-09-12T15:54:21.718 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-12T15:54:21.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T15:54:22.052 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:22.053 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T15:54:22.064 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-12T15:54:20.133951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95128,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020392,"statfs":{"total":289910292480,"available":289812881408,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.639815"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505765+0000","last_change":"2024-09-12T15:54:18.505765+0000","last_active":"2024-09-12T15:54:18.505765+0000","last_peered":"2024-09-12T15:54:18.505765+0000","last_clean":"2024-09-12T15:54:18.505765+0000","last_became_active":"2024-09-12T15:54:16.504253+0000","last_became_peered":"2024-09-12T15:54:16.504253+0000","last_unstale":"2024-09-12T15:54:18.505765+0000","last_undegraded":"2024-09-12T15:54:18.505765+0000","last_fullsized":"2024-09-12T15:54:18.505765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:38:53.254216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001172934,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505043+0000","last_change":"2024-09-12T15:54:18.505043+0000","last_active":"2024-09-12T15:54:18.505043+0000","last_peered":"2024-09-12T15:54:18.505043+0000","last_clean":"2024-09-12T15:54:18.505043+0000","last_became_active":"2024-09-12T15:54:16.504145+0000","last_became_peered":"2024-09-12T15:54:16.504145+0000","last_unstale":"2024-09-12T15:54:18.505043+0000","last_undegraded":"2024-09-12T15:54:18.505043+0000","last_fullsized":"2024-09-12T15:54:18.505043+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:15:43.919423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073703799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505427+0000","last_change":"2024-09-12T15:54:18.505427+0000","last_active":"2024-09-12T15:54:18.505427+0000","last_peered":"2024-09-12T15:54:18.505427+0000","last_clean":"2024-09-12T15:54:18.505427+0000","last_became_active":"2024-09-12T15:54:16.503301+0000","last_became_peered":"2024-09-12T15:54:16.503301+0000","last_unstale":"2024-09-12T15:54:18.505427+0000","last_undegraded":"2024-09-12T15:54:18.505427+0000","last_fullsized":"2024-09-12T15:54:18.505427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:26:05.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079584,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505017+0000","last_change":"2024-09-12T15:54:18.505017+0000","last_active":"2024-09-12T15:54:18.505017+0000","last_peered":"2024-09-12T15:54:18.505017+0000","last_clean":"2024-09-12T15:54:18.505017+0000","last_became_active":"2024-09-12T15:54:16.504202+0000","last_became_peered":"2024-09-12T15:54:16.504202+0000","last_unstale":"2024-09-12T15:54:18.505017+0000","last_undegraded":"2024-09-12T15:54:18.505017+0000","last_fullsized":"2024-09-12T15:54:18.505017+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T02:35:33.901895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007492,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.508822+0000","last_change":"2024-09-12T15:54:18.508120+0000","last_active":"2024-09-12T15:54:18.508822+0000","last_peered":"2024-09-12T15:54:18.508822+0000","last_clean":"2024-09-12T15:54:18.508822+0000","last_became_active":"2024-09-12T15:54:16.503357+0000","last_became_peered":"2024-09-12T15:54:16.503357+0000","last_unstale":"2024-09-12T15:54:18.508822+0000","last_undegraded":"2024-09-12T15:54:18.508822+0000","last_fullsized":"2024-09-12T15:54:18.508822+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:03:29.295197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001241416,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.830314+0000","last_change":"2024-09-12T15:54:18.830314+0000","last_active":"2024-09-12T15:54:18.830314+0000","last_peered":"2024-09-12T15:54:18.830314+0000","last_clean":"2024-09-12T15:54:18.830314+0000","last_became_active":"2024-09-12T15:54:16.505951+0000","last_became_peered":"2024-09-12T15:54:16.505951+0000","last_unstale":"2024-09-12T15:54:18.830314+0000","last_undegraded":"2024-09-12T15:54:18.830314+0000","last_fullsized":"2024-09-12T15:54:18.830314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:25:36.672104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039973199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.830367+0000","last_change":"2024-09-12T15:54:18.830367+0000","last_active":"2024-09-12T15:54:18.830367+0000","last_peered":"2024-09-12T15:54:18.830367+0000","last_clean":"2024-09-12T15:54:18.830367+0000","last_became_active":"2024-09-12T15:54:16.506142+0000","last_became_peered":"2024-09-12T15:54:16.506142+0000","last_unstale":"2024-09-12T15:54:18.830367+0000","last_undegraded":"2024-09-12T15:54:18.830367+0000","last_fullsized":"2024-09-12T15:54:18.830367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:22:17.633227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043801199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505434+0000","last_change":"2024-09-12T15:54:18.505434+0000","last_active":"2024-09-12T15:54:18.505434+0000","last_peered":"2024-09-12T15:54:18.505434+0000","last_clean":"2024-09-12T15:54:18.505434+0000","last_became_active":"2024-09-12T15:54:16.505397+0000","last_became_peered":"2024-09-12T15:54:16.505397+0000","last_unstale":"2024-09-12T15:54:18.505434+0000","last_undegraded":"2024-09-12T15:54:18.505434+0000","last_fullsized":"2024-09-12T15:54:18.505434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:48:51.220899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099554,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.504316+0000","last_change":"2024-09-12T15:54:11.528229+0000","last_active":"2024-09-12T15:54:18.504316+0000","last_peered":"2024-09-12T15:54:18.504316+0000","last_clean":"2024-09-12T15:54:18.504316+0000","last_became_active":"2024-09-12T15:54:11.527071+0000","last_became_peered":"2024-09-12T15:54:11.527071+0000","last_unstale":"2024-09-12T15:54:18.504316+0000","last_undegraded":"2024-09-12T15:54:18.504316+0000","last_fullsized":"2024-09-12T15:54:18.504316+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_clean_scrub_stamp":"2024-09-12T15:54:10.483728+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:21:13.353299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31900,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T15:54:22.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T15:54:22.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:22.361 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T15:54:22.373 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-12T15:54:22.134199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95128,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020392,"statfs":{"total":289910292480,"available":289812881408,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000982"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505765+0000","last_change":"2024-09-12T15:54:18.505765+0000","last_active":"2024-09-12T15:54:18.505765+0000","last_peered":"2024-09-12T15:54:18.505765+0000","last_clean":"2024-09-12T15:54:18.505765+0000","last_became_active":"2024-09-12T15:54:16.504253+0000","last_became_peered":"2024-09-12T15:54:16.504253+0000","last_unstale":"2024-09-12T15:54:18.505765+0000","last_undegraded":"2024-09-12T15:54:18.505765+0000","last_fullsized":"2024-09-12T15:54:18.505765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:38:53.254216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001172934,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505043+0000","last_change":"2024-09-12T15:54:18.505043+0000","last_active":"2024-09-12T15:54:18.505043+0000","last_peered":"2024-09-12T15:54:18.505043+0000","last_clean":"2024-09-12T15:54:18.505043+0000","last_became_active":"2024-09-12T15:54:16.504145+0000","last_became_peered":"2024-09-12T15:54:16.504145+0000","last_unstale":"2024-09-12T15:54:18.505043+0000","last_undegraded":"2024-09-12T15:54:18.505043+0000","last_fullsized":"2024-09-12T15:54:18.505043+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:15:43.919423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073703799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505427+0000","last_change":"2024-09-12T15:54:18.505427+0000","last_active":"2024-09-12T15:54:18.505427+0000","last_peered":"2024-09-12T15:54:18.505427+0000","last_clean":"2024-09-12T15:54:18.505427+0000","last_became_active":"2024-09-12T15:54:16.503301+0000","last_became_peered":"2024-09-12T15:54:16.503301+0000","last_unstale":"2024-09-12T15:54:18.505427+0000","last_undegraded":"2024-09-12T15:54:18.505427+0000","last_fullsized":"2024-09-12T15:54:18.505427+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:26:05.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079584,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505017+0000","last_change":"2024-09-12T15:54:18.505017+0000","last_active":"2024-09-12T15:54:18.505017+0000","last_peered":"2024-09-12T15:54:18.505017+0000","last_clean":"2024-09-12T15:54:18.505017+0000","last_became_active":"2024-09-12T15:54:16.504202+0000","last_became_peered":"2024-09-12T15:54:16.504202+0000","last_unstale":"2024-09-12T15:54:18.505017+0000","last_undegraded":"2024-09-12T15:54:18.505017+0000","last_fullsized":"2024-09-12T15:54:18.505017+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T02:35:33.901895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007492,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.508822+0000","last_change":"2024-09-12T15:54:18.508120+0000","last_active":"2024-09-12T15:54:18.508822+0000","last_peered":"2024-09-12T15:54:18.508822+0000","last_clean":"2024-09-12T15:54:18.508822+0000","last_became_active":"2024-09-12T15:54:16.503357+0000","last_became_peered":"2024-09-12T15:54:16.503357+0000","last_unstale":"2024-09-12T15:54:18.508822+0000","last_undegraded":"2024-09-12T15:54:18.508822+0000","last_fullsized":"2024-09-12T15:54:18.508822+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:03:29.295197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001241416,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.830314+0000","last_change":"2024-09-12T15:54:18.830314+0000","last_active":"2024-09-12T15:54:18.830314+0000","last_peered":"2024-09-12T15:54:18.830314+0000","last_clean":"2024-09-12T15:54:18.830314+0000","last_became_active":"2024-09-12T15:54:16.505951+0000","last_became_peered":"2024-09-12T15:54:16.505951+0000","last_unstale":"2024-09-12T15:54:18.830314+0000","last_undegraded":"2024-09-12T15:54:18.830314+0000","last_fullsized":"2024-09-12T15:54:18.830314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:25:36.672104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039973199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.830367+0000","last_change":"2024-09-12T15:54:18.830367+0000","last_active":"2024-09-12T15:54:18.830367+0000","last_peered":"2024-09-12T15:54:18.830367+0000","last_clean":"2024-09-12T15:54:18.830367+0000","last_became_active":"2024-09-12T15:54:16.506142+0000","last_became_peered":"2024-09-12T15:54:16.506142+0000","last_unstale":"2024-09-12T15:54:18.830367+0000","last_undegraded":"2024-09-12T15:54:18.830367+0000","last_fullsized":"2024-09-12T15:54:18.830367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:22:17.633227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043801199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.505434+0000","last_change":"2024-09-12T15:54:18.505434+0000","last_active":"2024-09-12T15:54:18.505434+0000","last_peered":"2024-09-12T15:54:18.505434+0000","last_clean":"2024-09-12T15:54:18.505434+0000","last_became_active":"2024-09-12T15:54:16.505397+0000","last_became_peered":"2024-09-12T15:54:16.505397+0000","last_unstale":"2024-09-12T15:54:18.505434+0000","last_undegraded":"2024-09-12T15:54:18.505434+0000","last_fullsized":"2024-09-12T15:54:18.505434+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:48:51.220899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099554,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-12T15:54:18.504316+0000","last_change":"2024-09-12T15:54:11.528229+0000","last_active":"2024-09-12T15:54:18.504316+0000","last_peered":"2024-09-12T15:54:18.504316+0000","last_clean":"2024-09-12T15:54:18.504316+0000","last_became_active":"2024-09-12T15:54:11.527071+0000","last_became_peered":"2024-09-12T15:54:11.527071+0000","last_unstale":"2024-09-12T15:54:18.504316+0000","last_undegraded":"2024-09-12T15:54:18.504316+0000","last_fullsized":"2024-09-12T15:54:18.504316+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_clean_scrub_stamp":"2024-09-12T15:54:10.483728+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:21:13.353299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31900,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T15:54:22.373 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-12T15:54:22.374 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-12T15:54:22.374 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-12T15:54:22.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-12T15:54:22.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:22.707 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-12T15:54:22.720 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-12T15:54:22.720 INFO:teuthology.run_tasks:Running task ceph... 2024-09-12T15:54:22.731 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-12T15:54:22.731 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-12T15:54:22.731 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7902659', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1680, 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'sleep_before_teardown': 0, 'subset': '42/100', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1', 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOGlv8KdgRy99aYpxFzc9OxbFOhe+Psd2fGj+dWHhm0eXiEVM74bK2aUhC+j8nwkTndUeu+LbLlyO70xx6cAFLA=', 'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnPQM/fgXjYviHXSJUTvWuMdxI2P0i0UhacxnQunjFI8ccRd23NudHOCUn715FX1le15xAEncHuHd6ajf78PJI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-12_15:21:08', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-12T15:54:22.732 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-12T15:54:22.734 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-12T15:54:22.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T15:54:22.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-09-12T15:54:22.825 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:22.825 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-09-12T15:54:22.884 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-12T15:54:22.884 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-09-12T15:54:22.942 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-12 15:53:31.444591913 +0000 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-12 15:51:01.380847614 +0000 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-12 15:51:01.380847614 +0000 2024-09-12T15:54:22.943 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-12 15:51:01.380847614 +0000 2024-09-12T15:54:22.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-12T15:54:23.013 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-12T15:54:23.013 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-12T15:54:23.013 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000221264 s, 2.3 MB/s 2024-09-12T15:54:23.015 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-12T15:54:23.072 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-12 15:53:31.445591905 +0000 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-12 15:51:01.711845016 +0000 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-12 15:51:01.711845016 +0000 2024-09-12T15:54:23.129 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-12 15:51:01.711845016 +0000 2024-09-12T15:54:23.130 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-12T15:54:23.193 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-12T15:54:23.193 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-12T15:54:23.193 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000197328 s, 2.6 MB/s 2024-09-12T15:54:23.194 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-12T15:54:23.251 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-12 15:53:31.445591905 +0000 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-12 15:51:02.033842488 +0000 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-12 15:51:02.033842488 +0000 2024-09-12T15:54:23.308 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-12 15:51:02.033842488 +0000 2024-09-12T15:54:23.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-12T15:54:23.372 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-12T15:54:23.372 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-12T15:54:23.372 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000192405 s, 2.7 MB/s 2024-09-12T15:54:23.373 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-12T15:54:23.431 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-09-12T15:54:23.488 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-12T15:54:23.488 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-12T15:54:23.488 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-12T15:54:23.488 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-12T15:54:23.488 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-09-12T15:54:23.488 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-09-12 15:53:31.446591896 +0000 2024-09-12T15:54:23.488 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-09-12 15:51:02.370839842 +0000 2024-09-12T15:54:23.489 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-09-12 15:51:02.370839842 +0000 2024-09-12T15:54:23.489 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-09-12 15:51:02.370839842 +0000 2024-09-12T15:54:23.489 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-12T15:54:23.553 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-09-12T15:54:23.553 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-09-12T15:54:23.554 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000214106 s, 2.4 MB/s 2024-09-12T15:54:23.555 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-12T15:54:23.611 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:54:23.611 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T15:54:23.611 INFO:tasks.ceph:Generating config... 2024-09-12T15:54:23.613 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-12T15:54:23.613 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client] log to stderr = False 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:23.614 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:23.614 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-12T15:54:23.614 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-12T15:54:23.614 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-12T15:54:23.615 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-12T15:54:23.615 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-12T15:54:23.615 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-12T15:54:23.615 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-12T15:54:23.615 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-12T15:54:23.615 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-12T15:54:23.616 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-12T15:54:23.616 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-12T15:54:23.616 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-12T15:54:23.616 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-12T15:54:23.616 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-12T15:54:23.616 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-12T15:54:23.616 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-12T15:54:23.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-12T15:54:23.694 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.keyring 2024-09-12T15:54:23.697 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-12T15:54:23.782 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-12T15:54:23.807 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.19')] 2024-09-12T15:54:23.808 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.19', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-12T15:54:23.809 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:23.809 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T15:54:23.864 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.19 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:23.945 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:23.945 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid c66699a9-7576-4b7c-b56e-924785d0fe22 2024-09-12T15:54:23.945 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = quincy 2024-09-12T15:54:23.945 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2024-09-12T15:54:23.945 INFO:teuthology.orchestra.run.smithi019.stdout:fsid c66699a9-7576-4b7c-b56e-924785d0fe22 2024-09-12T15:54:23.945 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2024-09-12T15:54:23.945178+0000 2024-09-12T15:54:23.945 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-09-12T15:54:23.945178+0000 2024-09-12T15:54:23.946 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 17 (quincy) 2024-09-12T15:54:23.946 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2024-09-12T15:54:23.946 INFO:teuthology.orchestra.run.smithi019.stdout:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-09-12T15:54:23.946 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-12T15:54:23.947 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-12T15:54:24.003 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID c66699a9-7576-4b7c-b56e-924785d0fe22... 2024-09-12T15:54:24.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-12T15:54:24.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-12T15:54:24.057 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-09-12T15:54:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-09-12T15:54:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-09-12T15:54:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = 172.21.15.19 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: mon client directed command retry = 5 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on pool no app = False 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject socket failures = 5000 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = c66699a9-7576-4b7c-b56e-924785d0fe22 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-09-12T15:54:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-09-12T15:54:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-09-12T15:54:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: osd memory target = 1610612736 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-09-12T15:54:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T15:54:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-09-12T15:54:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: debug rbd = 20 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: debug rbd_mirror = 15 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: log to stderr = False 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: rbd default features = 125 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: rbd mirror image policy type = none 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.0] 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.068 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.1] 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.2] 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout:[client.mirror.3] 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.088 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-09-12T15:54:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-09-12T15:54:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-09-12T15:54:24.090 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = 172.21.15.19 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: mon client directed command retry = 5 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on pool no app = False 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: ms inject socket failures = 5000 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = c66699a9-7576-4b7c-b56e-924785d0fe22 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-09-12T15:54:24.092 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.093 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: bdev async discard = True 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: bdev enable discard = True 2024-09-12T15:54:24.094 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 1/20 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 1/20 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 4/10 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: osd memory target = 1610612736 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2024-09-12T15:54:24.095 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-09-12T15:54:24.096 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.097 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-09-12T15:54:24.098 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd = 20 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd_mirror = 15 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: log to stderr = False 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: rbd default features = 125 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: rbd mirror image policy type = none 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.0] 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.1] 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.2] 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.100 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.3] 2024-09-12T15:54:24.100 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-12T15:54:24.100 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-12T15:54:24.100 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-12T15:54:24.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-12T15:54:24.157 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-12T15:54:24.157 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:24.157 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-12T15:54:24.173 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:24.173 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-12T15:54:24.228 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T15:54:24.228 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:54:24.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-12T15:54:24.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-12T15:54:24.268 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:54:24.268 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:24.326 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T15:54:24.326 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:24.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-12T15:54:24.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-12T15:54:24.363 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:24.363 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:24.420 INFO:tasks.ceph:Setting up mon nodes... 2024-09-12T15:54:24.420 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-12T15:54:24.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-12T15:54:24.476 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-12T15:54:24.479 INFO:tasks.ceph:Setting up mds nodes... 2024-09-12T15:54:24.479 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-12T15:54:24.479 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-12T15:54:24.523 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-12T15:54:24.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-12T15:54:24.579 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-12T15:54:24.593 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-12T15:54:24.675 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-12T15:54:24.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-12T15:54:24.771 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-12T15:54:24.784 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-12T15:54:24.866 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-12T15:54:24.879 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-12T15:54:24.963 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-12T15:54:24.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-12T15:54:25.058 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-12T15:54:25.071 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-12T15:54:25.153 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-12T15:54:25.166 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-12T15:54:25.248 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-12T15:54:25.261 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-12T15:54:25.261 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-12T15:54:25.262 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-12T15:54:25.327 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:54:25.327 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-12T15:54:25.327 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T15:54:25.327 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T15:54:25.400 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T15:54:25.906 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-09-12T15:54:25.909 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-09-12T15:54:25.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-12T15:54:26.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-12T15:54:26.042 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-12T15:54:26.108 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:54:26.109 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-12T15:54:26.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T15:54:26.109 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-12T15:54:26.182 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T15:54:26.183 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T15:54:26.638 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-09-12T15:54:26.640 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-09-12T15:54:26.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-12T15:54:26.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-12T15:54:26.789 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-12T15:54:26.857 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-12T15:54:26.857 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-12T15:54:26.857 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T15:54:26.857 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-12T15:54:26.932 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-12T15:54:26.932 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-12T15:54:26.932 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-12T15:54:26.933 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-12T15:54:26.933 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-12T15:54:26.933 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-09-12T15:54:26.933 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-12T15:54:26.933 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-12T15:54:26.933 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-12T15:54:26.933 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-12T15:54:27.431 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-09-12T15:54:27.433 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-09-12T15:54:27.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-12T15:54:27.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-12T15:54:27.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:27.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:27.658+0000 7f3c8641f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-12T15:54:27.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:27.658+0000 7f3c8641f640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-12T15:54:27.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:27.658+0000 7f3c8641f640 -1 bdev(0x5562f052d800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-12T15:54:27.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:27.658+0000 7f3c8641f640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-12T15:54:31.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-12T15:54:31.094 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:31.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:31.185+0000 7f0f81033640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-12T15:54:31.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:31.185+0000 7f0f81033640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-12T15:54:31.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:31.186+0000 7f0f81033640 -1 bdev(0x556000c1d800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-12T15:54:31.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:31.186+0000 7f0f81033640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-12T15:54:34.350 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-12T15:54:34.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:34.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:34.473+0000 7fe8e3415640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-12T15:54:34.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:34.473+0000 7fe8e3415640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-12T15:54:34.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:34.473+0000 7fe8e3415640 -1 bdev(0x56069673d800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-12T15:54:34.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T15:54:34.473+0000 7fe8e3415640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-12T15:54:37.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-12T15:54:37.411 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-12T15:54:37.411 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:54:37.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-12T15:54:37.443 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.443 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-12T15:54:37.481 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.481 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-12T15:54:37.551 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-12T15:54:37.620 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-12T15:54:37.620 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-12T15:54:37.640 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.640 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-12T15:54:37.678 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.678 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-12T15:54:37.735 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.735 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-12T15:54:37.793 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.793 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-12T15:54:37.850 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.850 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-12T15:54:37.909 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.909 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-12T15:54:37.965 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:37.965 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-12T15:54:38.023 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:38.023 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-12T15:54:38.079 INFO:tasks.ceph:Adding keys to all mons... 2024-09-12T15:54:38.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-12T15:54:38.145 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2024-09-12T15:54:38.145 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNmpDVpHBAA3gL+lA5NlPlv6jPHCf2urw== 2024-09-12T15:54:38.145 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-09-12T15:54:38.145 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCzDuNmoa9JJxAAxPAycodOyOQqJcaoCskYKg== 2024-09-12T15:54:38.145 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-09-12T15:54:38.145 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQC3DuNmLP8eCxAA2xcqNPrz5BlCetIJfOd2XA== 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQC6DuNmgzM/HBAAvjcPOF8QNwhf3euKWqkONg== 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNmY+Q0HxAAN87CCnCPW8fn2/0F9q8cHQ== 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror] 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNm08GQIhAAfTwyY3subm/29e7JsDQtaQ== 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.0] 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNmHCxJKBAAjmTooErtXIxM4Sg577Qdfg== 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.1] 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNm96j3LRAAd+1A+9ZLGWDdywBOBBIHTg== 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.2] 2024-09-12T15:54:38.146 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNmuTmjMxAA4i9GvL13l9ieWOTkni9cFw== 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.3] 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNm9/5mORAAE+ekh9NBEblHOnx9Yl7Rmw== 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.4] 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCxDuNm8jl7AxAALnAARunR6+DuwCzrpIBQhg== 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.5] 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCxDuNm8XEkCRAAqkNWQErXWhMCpo3Kw8Y8LQ== 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.6] 2024-09-12T15:54:38.147 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCxDuNmDAnUDhAAdsamIipDhDgsP4CvrKNzbg== 2024-09-12T15:54:38.148 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-12T15:54:38.243 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T15:54:38.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T15:54:38.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-12T15:54:38.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.457 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.512 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.568 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.624 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.735 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.791 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-12T15:54:38.902 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-12T15:54:38.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-12T15:54:38.928 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-12T15:54:39.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-12T15:54:39.052 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-12T15:54:39.107 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-12T15:54:39.108 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-12T15:54:39.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-12T15:54:39.150 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-12T15:54:39.150 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-12T15:54:39.151 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-12T15:54:39.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-12T15:54:39.153 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-12T15:54:39.154 DEBUG:tasks.ceph:set 0 configs 2024-09-12T15:54:39.154 INFO:tasks.ceph:Setting crush tunables to default 2024-09-12T15:54:39.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-12T15:54:39.329 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-09-12T15:54:39.340 INFO:tasks.ceph:check_enable_crimson: False 2024-09-12T15:54:39.340 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-12T15:54:39.341 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:39.341 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-12T15:54:39.367 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:39.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-12T15:54:39.432 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:54:39.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-12T15:54:39.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster cluster2 osd new a737e4c9-23d4-49a4-b3aa-ea598ebb9e7f 0 2024-09-12T15:54:39.721 INFO:teuthology.orchestra.run.smithi019.stdout:0 2024-09-12T15:54:39.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster cluster2 osd new d2c02160-4cc3-4bbe-91aa-f51be5a2240e 1 2024-09-12T15:54:39.912 INFO:teuthology.orchestra.run.smithi019.stdout:1 2024-09-12T15:54:39.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster cluster2 osd new 2b0b4245-cf55-41c4-bf1e-3c3edb29d3cb 2 2024-09-12T15:54:40.103 INFO:teuthology.orchestra.run.smithi019.stdout:2 2024-09-12T15:54:40.114 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-12T15:54:40.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-12T15:54:40.116 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-12T15:54:40.117 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-12T15:54:40.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-12T15:54:40.118 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-12T15:54:40.119 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-12T15:54:40.119 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-12T15:54:40.121 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-12T15:54:40.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-12T15:54:40.297 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:40.298 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","created":"2024-09-12T15:54:39.237275+0000","modified":"2024-09-12T15:54:40.102697+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T15:54:40.102697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a737e4c9-23d4-49a4-b3aa-ea598ebb9e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2c02160-4cc3-4bbe-91aa-f51be5a2240e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b0b4245-cf55-41c4-bf1e-3c3edb29d3cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:40.308 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-12T15:54:40.308 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-12T15:54:40.611 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-12T15:54:40.774 INFO:teuthology.misc.health.smithi019.stdout: 2024-09-12T15:54:40.775 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":5,"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","created":"2024-09-12T15:54:39.237275+0000","modified":"2024-09-12T15:54:40.102697+0000","last_up_change":"0.000000","last_in_change":"2024-09-12T15:54:40.102697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a737e4c9-23d4-49a4-b3aa-ea598ebb9e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2c02160-4cc3-4bbe-91aa-f51be5a2240e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b0b4245-cf55-41c4-bf1e-3c3edb29d3cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:40.784 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-12T15:54:41.320 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T15:54:41.318+0000 7f767002f640 -1 Falling back to public interface 2024-09-12T15:54:41.326 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T15:54:41.324+0000 7ff709219640 -1 Falling back to public interface 2024-09-12T15:54:41.330 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T15:54:41.328+0000 7fbc6b831640 -1 Falling back to public interface 2024-09-12T15:54:44.079 INFO:tasks.ceph.cluster2.mgr.x.smithi016.stderr:2024-09-12T15:54:44.077+0000 7f90730d0640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-12T15:54:44.816 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T15:54:44.814+0000 7f767002f640 -1 osd.2 0 log_to_monitors true 2024-09-12T15:54:45.149 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T15:54:45.147+0000 7fbc6b831640 -1 osd.0 0 log_to_monitors true 2024-09-12T15:54:45.370 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T15:54:45.368+0000 7ff709219640 -1 osd.1 0 log_to_monitors true 2024-09-12T15:54:46.808 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T15:54:46.806+0000 7f766bfbe640 -1 osd.2 0 waiting for initial osdmap 2024-09-12T15:54:46.823 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T15:54:46.821+0000 7f7663ddd640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T15:54:47.087 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-12T15:54:47.416 INFO:teuthology.misc.health.smithi019.stdout: 2024-09-12T15:54:47.417 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":8,"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","created":"2024-09-12T15:54:39.237275+0000","modified":"2024-09-12T15:54:46.874706+0000","last_up_change":"2024-09-12T15:54:46.874706+0000","last_in_change":"2024-09-12T15:54:40.102697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a737e4c9-23d4-49a4-b3aa-ea598ebb9e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2c02160-4cc3-4bbe-91aa-f51be5a2240e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b0b4245-cf55-41c4-bf1e-3c3edb29d3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6801","nonce":2578076338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6803","nonce":2578076338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6807","nonce":2578076338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6805","nonce":2578076338}]},"public_addr":"172.21.15.19:6801/2578076338","cluster_addr":"172.21.15.19:6803/2578076338","heartbeat_back_addr":"172.21.15.19:6807/2578076338","heartbeat_front_addr":"172.21.15.19:6805/2578076338","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:47.431 DEBUG:teuthology.misc:1 of 3 OSDs are up 2024-09-12T15:54:48.313 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T15:54:48.310+0000 7fbc66d85640 -1 osd.0 0 waiting for initial osdmap 2024-09-12T15:54:48.317 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T15:54:48.315+0000 7ff7051a8640 -1 osd.1 0 waiting for initial osdmap 2024-09-12T15:54:48.326 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T15:54:48.324+0000 7fbc5eba4640 -1 osd.0 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T15:54:48.329 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T15:54:48.328+0000 7ff6fcfc7640 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-12T15:54:53.734 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-12T15:54:54.025 INFO:teuthology.misc.health.smithi019.stdout: 2024-09-12T15:54:54.026 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":14,"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","created":"2024-09-12T15:54:39.237275+0000","modified":"2024-09-12T15:54:53.972660+0000","last_up_change":"2024-09-12T15:54:49.318145+0000","last_in_change":"2024-09-12T15:54:40.102697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T15:54:51.035541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a737e4c9-23d4-49a4-b3aa-ea598ebb9e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6817","nonce":3591204202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6819","nonce":3591204202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6823","nonce":3591204202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6821","nonce":3591204202}]},"public_addr":"172.21.15.19:6817/3591204202","cluster_addr":"172.21.15.19:6819/3591204202","heartbeat_back_addr":"172.21.15.19:6823/3591204202","heartbeat_front_addr":"172.21.15.19:6821/3591204202","state":["exists","up"]},{"osd":1,"uuid":"d2c02160-4cc3-4bbe-91aa-f51be5a2240e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6809","nonce":2526212325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6811","nonce":2526212325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6815","nonce":2526212325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6813","nonce":2526212325}]},"public_addr":"172.21.15.19:6809/2526212325","cluster_addr":"172.21.15.19:6811/2526212325","heartbeat_back_addr":"172.21.15.19:6815/2526212325","heartbeat_front_addr":"172.21.15.19:6813/2526212325","state":["exists","up"]},{"osd":2,"uuid":"2b0b4245-cf55-41c4-bf1e-3c3edb29d3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6801","nonce":2578076338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6803","nonce":2578076338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6807","nonce":2578076338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6805","nonce":2578076338}]},"public_addr":"172.21.15.19:6801/2578076338","cluster_addr":"172.21.15.19:6803/2578076338","heartbeat_back_addr":"172.21.15.19:6807/2578076338","heartbeat_front_addr":"172.21.15.19:6805/2578076338","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:46.152985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:46.335509+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:45.782123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:54.037 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-12T15:54:54.037 INFO:tasks.ceph:Creating RBD pool 2024-09-12T15:54:54.037 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-12T15:54:54.983 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'rbd' created 2024-09-12T15:54:55.001 DEBUG:teuthology.orchestra.run.smithi019:> rbd --cluster cluster2 pool init rbd 2024-09-12T15:54:58.000 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-12T15:54:58.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 config log 1 --format=json 2024-09-12T15:54:58.001 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-12T15:54:58.309 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:58.321 INFO:teuthology.orchestra.run.smithi019.stdout:[{"version":4,"timestamp":"2024-09-12T15:54:48.317719+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"5060.617143"}]}] 2024-09-12T15:54:58.322 INFO:tasks.ceph_manager:config epoch is 4 2024-09-12T15:54:58.322 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-12T15:54:58.322 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-12T15:54:58.322 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-12T15:54:58.671 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:58.682 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":3943199050},{"type":"v1","addr":"172.21.15.16:6827","nonce":3943199050}]},"active_addr":"172.21.15.16:6827/3943199050","active_change":"2024-09-12T15:54:42.812518+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2308458044}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1848599653}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":718453239}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":606745960}]}]} 2024-09-12T15:54:58.684 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-12T15:54:58.685 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-12T15:54:58.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-12T15:54:58.985 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:58.986 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":18,"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","created":"2024-09-12T15:54:39.237275+0000","modified":"2024-09-12T15:54:57.988699+0000","last_up_change":"2024-09-12T15:54:49.318145+0000","last_in_change":"2024-09-12T15:54:40.102697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T15:54:51.035541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T15:54:54.339805+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a737e4c9-23d4-49a4-b3aa-ea598ebb9e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6817","nonce":3591204202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6819","nonce":3591204202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6823","nonce":3591204202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6821","nonce":3591204202}]},"public_addr":"172.21.15.19:6817/3591204202","cluster_addr":"172.21.15.19:6819/3591204202","heartbeat_back_addr":"172.21.15.19:6823/3591204202","heartbeat_front_addr":"172.21.15.19:6821/3591204202","state":["exists","up"]},{"osd":1,"uuid":"d2c02160-4cc3-4bbe-91aa-f51be5a2240e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6809","nonce":2526212325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6811","nonce":2526212325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6815","nonce":2526212325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6813","nonce":2526212325}]},"public_addr":"172.21.15.19:6809/2526212325","cluster_addr":"172.21.15.19:6811/2526212325","heartbeat_back_addr":"172.21.15.19:6815/2526212325","heartbeat_front_addr":"172.21.15.19:6813/2526212325","state":["exists","up"]},{"osd":2,"uuid":"2b0b4245-cf55-41c4-bf1e-3c3edb29d3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6801","nonce":2578076338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6803","nonce":2578076338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6807","nonce":2578076338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6805","nonce":2578076338}]},"public_addr":"172.21.15.19:6801/2578076338","cluster_addr":"172.21.15.19:6803/2578076338","heartbeat_back_addr":"172.21.15.19:6807/2578076338","heartbeat_front_addr":"172.21.15.19:6805/2578076338","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:46.152985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:46.335509+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:45.782123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:58.998 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-12T15:54:58.999 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-12T15:54:59.296 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:54:59.296 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":18,"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","created":"2024-09-12T15:54:39.237275+0000","modified":"2024-09-12T15:54:57.988699+0000","last_up_change":"2024-09-12T15:54:49.318145+0000","last_in_change":"2024-09-12T15:54:40.102697+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T15:54:51.035541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T15:54:54.339805+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a737e4c9-23d4-49a4-b3aa-ea598ebb9e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6817","nonce":3591204202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6819","nonce":3591204202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6823","nonce":3591204202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":3591204202},{"type":"v1","addr":"172.21.15.19:6821","nonce":3591204202}]},"public_addr":"172.21.15.19:6817/3591204202","cluster_addr":"172.21.15.19:6819/3591204202","heartbeat_back_addr":"172.21.15.19:6823/3591204202","heartbeat_front_addr":"172.21.15.19:6821/3591204202","state":["exists","up"]},{"osd":1,"uuid":"d2c02160-4cc3-4bbe-91aa-f51be5a2240e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6809","nonce":2526212325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6811","nonce":2526212325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6815","nonce":2526212325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2526212325},{"type":"v1","addr":"172.21.15.19:6813","nonce":2526212325}]},"public_addr":"172.21.15.19:6809/2526212325","cluster_addr":"172.21.15.19:6811/2526212325","heartbeat_back_addr":"172.21.15.19:6815/2526212325","heartbeat_front_addr":"172.21.15.19:6813/2526212325","state":["exists","up"]},{"osd":2,"uuid":"2b0b4245-cf55-41c4-bf1e-3c3edb29d3cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6801","nonce":2578076338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6803","nonce":2578076338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6807","nonce":2578076338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2578076338},{"type":"v1","addr":"172.21.15.19:6805","nonce":2578076338}]},"public_addr":"172.21.15.19:6801/2578076338","cluster_addr":"172.21.15.19:6803/2578076338","heartbeat_back_addr":"172.21.15.19:6807/2578076338","heartbeat_front_addr":"172.21.15.19:6805/2578076338","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:46.152985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:46.335509+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:45.782123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T15:54:59.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-12T15:54:59.310 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-12T15:54:59.310 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-12T15:54:59.452 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-09-12T15:54:59.452 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-12T15:54:59.469 INFO:teuthology.orchestra.run.smithi019.stdout:42949672966 2024-09-12T15:54:59.469 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-12T15:54:59.470 INFO:teuthology.orchestra.run.smithi019.stdout:42949672966 2024-09-12T15:54:59.471 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-12T15:54:59.779 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2024-09-12T15:54:59.792 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-12T15:54:59.820 INFO:teuthology.orchestra.run.smithi019.stdout:42949672965 2024-09-12T15:54:59.832 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672966 got 42949672965 for osd.0 2024-09-12T15:54:59.832 INFO:teuthology.orchestra.run.smithi019.stdout:42949672965 2024-09-12T15:54:59.845 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672966 got 42949672965 for osd.1 2024-09-12T15:55:00.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-12T15:55:00.833 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-12T15:55:00.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-12T15:55:01.122 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2024-09-12T15:55:01.134 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2024-09-12T15:55:01.134 DEBUG:teuthology.parallel:result is None 2024-09-12T15:55:01.175 INFO:teuthology.orchestra.run.smithi019.stdout:42949672967 2024-09-12T15:55:01.187 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672966 got 42949672967 for osd.0 2024-09-12T15:55:01.187 DEBUG:teuthology.parallel:result is None 2024-09-12T15:55:01.210 INFO:teuthology.orchestra.run.smithi019.stdout:42949672967 2024-09-12T15:55:01.223 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672966 got 42949672967 for osd.1 2024-09-12T15:55:01.223 DEBUG:teuthology.parallel:result is None 2024-09-12T15:55:01.223 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-12T15:55:01.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T15:55:01.556 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:55:01.557 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T15:55:01.568 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-12T15:55:00.820416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82720,"kb_used_data":1624,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032800,"statfs":{"total":289910292480,"available":289825587200,"internally_reserved":0,"allocated":1662976,"data_stored":1315717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.837024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.995861+0000","last_change":"2024-09-12T15:54:57.995861+0000","last_active":"2024-09-12T15:54:57.995861+0000","last_peered":"2024-09-12T15:54:57.995861+0000","last_clean":"2024-09-12T15:54:57.995861+0000","last_became_active":"2024-09-12T15:54:55.994009+0000","last_became_peered":"2024-09-12T15:54:55.994009+0000","last_unstale":"2024-09-12T15:54:57.995861+0000","last_undegraded":"2024-09-12T15:54:57.995861+0000","last_fullsized":"2024-09-12T15:54:57.995861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:49.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.996101+0000","last_change":"2024-09-12T15:54:57.996101+0000","last_active":"2024-09-12T15:54:57.996101+0000","last_peered":"2024-09-12T15:54:57.996101+0000","last_clean":"2024-09-12T15:54:57.996101+0000","last_became_active":"2024-09-12T15:54:55.993510+0000","last_became_peered":"2024-09-12T15:54:55.993510+0000","last_unstale":"2024-09-12T15:54:57.996101+0000","last_undegraded":"2024-09-12T15:54:57.996101+0000","last_fullsized":"2024-09-12T15:54:57.996101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:45:05.287258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.996206+0000","last_change":"2024-09-12T15:54:57.996206+0000","last_active":"2024-09-12T15:54:57.996206+0000","last_peered":"2024-09-12T15:54:57.996206+0000","last_clean":"2024-09-12T15:54:57.996206+0000","last_became_active":"2024-09-12T15:54:55.994138+0000","last_became_peered":"2024-09-12T15:54:55.994138+0000","last_unstale":"2024-09-12T15:54:57.996206+0000","last_undegraded":"2024-09-12T15:54:57.996206+0000","last_fullsized":"2024-09-12T15:54:57.996206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:37:28.425081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.995564+0000","last_change":"2024-09-12T15:54:57.995564+0000","last_active":"2024-09-12T15:54:57.995564+0000","last_peered":"2024-09-12T15:54:57.995564+0000","last_clean":"2024-09-12T15:54:57.995564+0000","last_became_active":"2024-09-12T15:54:55.994013+0000","last_became_peered":"2024-09-12T15:54:55.994013+0000","last_unstale":"2024-09-12T15:54:57.995564+0000","last_undegraded":"2024-09-12T15:54:57.995564+0000","last_fullsized":"2024-09-12T15:54:57.995564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:45:32.048846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.993807+0000","last_change":"2024-09-12T15:54:57.991815+0000","last_active":"2024-09-12T15:54:57.993807+0000","last_peered":"2024-09-12T15:54:57.993807+0000","last_clean":"2024-09-12T15:54:57.993807+0000","last_became_active":"2024-09-12T15:54:55.995684+0000","last_became_peered":"2024-09-12T15:54:55.995684+0000","last_unstale":"2024-09-12T15:54:57.993807+0000","last_undegraded":"2024-09-12T15:54:57.993807+0000","last_fullsized":"2024-09-12T15:54:57.993807+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:34:28.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:58.074588+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T15:54:58.074588+0000","last_peered":"2024-09-12T15:54:58.074588+0000","last_clean":"2024-09-12T15:54:58.074588+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T15:54:58.074588+0000","last_undegraded":"2024-09-12T15:54:58.074588+0000","last_fullsized":"2024-09-12T15:54:58.074588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:58.074534+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T15:54:58.074534+0000","last_peered":"2024-09-12T15:54:58.074534+0000","last_clean":"2024-09-12T15:54:58.074534+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T15:54:58.074534+0000","last_undegraded":"2024-09-12T15:54:58.074534+0000","last_fullsized":"2024-09-12T15:54:58.074534+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.995557+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T15:54:57.995557+0000","last_peered":"2024-09-12T15:54:57.995557+0000","last_clean":"2024-09-12T15:54:57.995557+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T15:54:57.995557+0000","last_undegraded":"2024-09-12T15:54:57.995557+0000","last_fullsized":"2024-09-12T15:54:57.995557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'52","reported_seq":88,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.994817+0000","last_change":"2024-09-12T15:54:53.174652+0000","last_active":"2024-09-12T15:54:57.994817+0000","last_peered":"2024-09-12T15:54:57.994817+0000","last_clean":"2024-09-12T15:54:57.994817+0000","last_became_active":"2024-09-12T15:54:53.173576+0000","last_became_peered":"2024-09-12T15:54:53.173576+0000","last_unstale":"2024-09-12T15:54:57.994817+0000","last_undegraded":"2024-09-12T15:54:57.994817+0000","last_fullsized":"2024-09-12T15:54:57.994817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:54:07.202748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":753664,"data_stored":635368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672967,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":635368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":155648,"data_stored":44981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T15:55:01.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T15:55:01.864 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:55:01.864 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T15:55:01.876 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-12T15:55:00.820416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82720,"kb_used_data":1624,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032800,"statfs":{"total":289910292480,"available":289825587200,"internally_reserved":0,"allocated":1662976,"data_stored":1315717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.837024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.995861+0000","last_change":"2024-09-12T15:54:57.995861+0000","last_active":"2024-09-12T15:54:57.995861+0000","last_peered":"2024-09-12T15:54:57.995861+0000","last_clean":"2024-09-12T15:54:57.995861+0000","last_became_active":"2024-09-12T15:54:55.994009+0000","last_became_peered":"2024-09-12T15:54:55.994009+0000","last_unstale":"2024-09-12T15:54:57.995861+0000","last_undegraded":"2024-09-12T15:54:57.995861+0000","last_fullsized":"2024-09-12T15:54:57.995861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:47:49.224346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.996101+0000","last_change":"2024-09-12T15:54:57.996101+0000","last_active":"2024-09-12T15:54:57.996101+0000","last_peered":"2024-09-12T15:54:57.996101+0000","last_clean":"2024-09-12T15:54:57.996101+0000","last_became_active":"2024-09-12T15:54:55.993510+0000","last_became_peered":"2024-09-12T15:54:55.993510+0000","last_unstale":"2024-09-12T15:54:57.996101+0000","last_undegraded":"2024-09-12T15:54:57.996101+0000","last_fullsized":"2024-09-12T15:54:57.996101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:45:05.287258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.996206+0000","last_change":"2024-09-12T15:54:57.996206+0000","last_active":"2024-09-12T15:54:57.996206+0000","last_peered":"2024-09-12T15:54:57.996206+0000","last_clean":"2024-09-12T15:54:57.996206+0000","last_became_active":"2024-09-12T15:54:55.994138+0000","last_became_peered":"2024-09-12T15:54:55.994138+0000","last_unstale":"2024-09-12T15:54:57.996206+0000","last_undegraded":"2024-09-12T15:54:57.996206+0000","last_fullsized":"2024-09-12T15:54:57.996206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:37:28.425081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.995564+0000","last_change":"2024-09-12T15:54:57.995564+0000","last_active":"2024-09-12T15:54:57.995564+0000","last_peered":"2024-09-12T15:54:57.995564+0000","last_clean":"2024-09-12T15:54:57.995564+0000","last_became_active":"2024-09-12T15:54:55.994013+0000","last_became_peered":"2024-09-12T15:54:55.994013+0000","last_unstale":"2024-09-12T15:54:57.995564+0000","last_undegraded":"2024-09-12T15:54:57.995564+0000","last_fullsized":"2024-09-12T15:54:57.995564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:45:32.048846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.993807+0000","last_change":"2024-09-12T15:54:57.991815+0000","last_active":"2024-09-12T15:54:57.993807+0000","last_peered":"2024-09-12T15:54:57.993807+0000","last_clean":"2024-09-12T15:54:57.993807+0000","last_became_active":"2024-09-12T15:54:55.995684+0000","last_became_peered":"2024-09-12T15:54:55.995684+0000","last_unstale":"2024-09-12T15:54:57.993807+0000","last_undegraded":"2024-09-12T15:54:57.993807+0000","last_fullsized":"2024-09-12T15:54:57.993807+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:34:28.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:58.074588+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T15:54:58.074588+0000","last_peered":"2024-09-12T15:54:58.074588+0000","last_clean":"2024-09-12T15:54:58.074588+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T15:54:58.074588+0000","last_undegraded":"2024-09-12T15:54:58.074588+0000","last_fullsized":"2024-09-12T15:54:58.074588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:58.074534+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T15:54:58.074534+0000","last_peered":"2024-09-12T15:54:58.074534+0000","last_clean":"2024-09-12T15:54:58.074534+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T15:54:58.074534+0000","last_undegraded":"2024-09-12T15:54:58.074534+0000","last_fullsized":"2024-09-12T15:54:58.074534+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.995557+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T15:54:57.995557+0000","last_peered":"2024-09-12T15:54:57.995557+0000","last_clean":"2024-09-12T15:54:57.995557+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T15:54:57.995557+0000","last_undegraded":"2024-09-12T15:54:57.995557+0000","last_fullsized":"2024-09-12T15:54:57.995557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'52","reported_seq":88,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-12T15:54:57.994817+0000","last_change":"2024-09-12T15:54:53.174652+0000","last_active":"2024-09-12T15:54:57.994817+0000","last_peered":"2024-09-12T15:54:57.994817+0000","last_clean":"2024-09-12T15:54:57.994817+0000","last_became_active":"2024-09-12T15:54:53.173576+0000","last_became_peered":"2024-09-12T15:54:53.173576+0000","last_unstale":"2024-09-12T15:54:57.994817+0000","last_undegraded":"2024-09-12T15:54:57.994817+0000","last_fullsized":"2024-09-12T15:54:57.994817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:54:07.202748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":753664,"data_stored":635368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672967,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":753664,"data_stored":635368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":155648,"data_stored":44981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T15:55:01.877 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-12T15:55:01.877 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-12T15:55:01.877 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-12T15:55:01.878 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-12T15:55:02.210 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T15:55:02.210 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-12T15:55:02.223 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-12T15:55:02.223 INFO:teuthology.run_tasks:Running task exec... 2024-09-12T15:55:02.234 INFO:teuthology.task.exec:Executing custom commands... 2024-09-12T15:55:02.234 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T15:55:02.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:02.611 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror] 2024-09-12T15:55:02.611 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCKDuNmMCJwLxAAy//QuVBPt6cMhxoXt8TDEw== 2024-09-12T15:55:02.611 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-12T15:55:02.611 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:02.611 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror 2024-09-12T15:55:02.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:03.007 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.0] 2024-09-12T15:55:03.008 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCKDuNmmy/oMhAAtIh13R2FYm/iIOhWdMqX8A== 2024-09-12T15:55:03.008 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:03.008 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:03.008 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.0 2024-09-12T15:55:03.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:03.405 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.1] 2024-09-12T15:55:03.405 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCKDuNmSkU+NhAAVaYrusdKgeLRjijV+7SpzA== 2024-09-12T15:55:03.406 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:03.406 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:03.406 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.1 2024-09-12T15:55:03.419 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:03.805 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.2] 2024-09-12T15:55:03.805 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCLDuNmBxthABAAo7+yHGhe7Q7jpPvQqbgwag== 2024-09-12T15:55:03.805 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:03.805 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:03.806 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.2 2024-09-12T15:55:03.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:04.200 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.3] 2024-09-12T15:55:04.200 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCLDuNmRmgaBhAAE9Rm6nGFw62PMkzau6B2lA== 2024-09-12T15:55:04.200 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:04.200 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:04.201 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.3 2024-09-12T15:55:04.213 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:04.594 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror] 2024-09-12T15:55:04.594 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNm08GQIhAAfTwyY3subm/29e7JsDQtaQ== 2024-09-12T15:55:04.595 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-12T15:55:04.595 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:04.595 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror 2024-09-12T15:55:04.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:04.994 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.0] 2024-09-12T15:55:04.994 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNmHCxJKBAAjmTooErtXIxM4Sg577Qdfg== 2024-09-12T15:55:04.994 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:04.995 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:04.995 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.0 2024-09-12T15:55:05.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:05.390 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.1] 2024-09-12T15:55:05.390 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNm96j3LRAAd+1A+9ZLGWDdywBOBBIHTg== 2024-09-12T15:55:05.390 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:05.390 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:05.391 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.1 2024-09-12T15:55:05.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:05.785 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.2] 2024-09-12T15:55:05.785 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNmuTmjMxAA4i9GvL13l9ieWOTkni9cFw== 2024-09-12T15:55:05.785 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:05.785 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:05.786 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.2 2024-09-12T15:55:05.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-12T15:55:06.180 INFO:teuthology.orchestra.run.smithi019.stdout:[client.mirror.3] 2024-09-12T15:55:06.180 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCwDuNm9/5mORAAE+ekh9NBEblHOnx9Yl7Rmw== 2024-09-12T15:55:06.180 INFO:teuthology.orchestra.run.smithi019.stdout: caps mon = "profile rbd-mirror" 2024-09-12T15:55:06.180 INFO:teuthology.orchestra.run.smithi019.stdout: caps osd = "profile rbd" 2024-09-12T15:55:06.181 INFO:teuthology.orchestra.run.smithi019.stderr:updated caps for client.mirror.3 2024-09-12T15:55:06.194 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T15:55:06.205 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T15:55:06.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T15:55:06.207 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-12T15:55:06.216 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-12T15:55:06.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-12T15:55:06.218 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-12T15:55:06.227 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-12T15:55:06.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-12T15:55:06.229 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-12T15:55:06.238 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-12T15:55:06.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-12T15:55:06.240 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-12T15:55:06.250 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-12T15:55:06.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-12T15:55:06.253 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-12T15:55:06.263 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-12T15:55:06.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-12T15:55:06.265 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-12T15:55:06.275 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-12T15:55:06.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-12T15:55:06.278 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-12T15:55:06.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-12T15:55:06.286 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-12T15:55:06.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-12T15:55:06.289 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-12T15:55:06.300 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726156506 2024-09-12T15:55:06.300 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-12T15:55:06.300 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-12T15:55:06.309 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726156506 2024-09-12T15:55:06.309 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-12T15:55:06.310 INFO:teuthology.run_tasks:Running task workunit... 2024-09-12T15:55:06.323 INFO:tasks.workunit:Pulling workunits from ref d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T15:55:06.323 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-12T15:55:06.323 DEBUG:teuthology.orchestra.run.smithi019:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T15:55:06.324 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-12T15:55:06.324 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-09-12T15:55:06.324 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-12T15:55:06.324 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-12T15:55:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T15:55:06.343 INFO:teuthology.orchestra.run.smithi019.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-12T15:55:06.344 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T15:55:06.400 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T15:55:06.401 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-12T15:55:06.457 INFO:tasks.workunit:timeout=6h 2024-09-12T15:55:06.457 INFO:tasks.workunit:cleanup=True 2024-09-12T15:55:06.458 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d61b08865d4d98738f45ff2e4806d7fb6d03eca1 2024-09-12T15:55:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-12T15:55:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Updating files: 97% (11734/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-09-12T15:55:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Note: switching to 'd61b08865d4d98738f45ff2e4806d7fb6d03eca1'. 2024-09-12T15:55:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:state without impacting any branches by switching back to a branch. 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: git switch -c 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Or undo this operation with: 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: git switch - 2024-09-12T15:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:55:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-12T15:55:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:55:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:HEAD is now at d61b08865d4 Merge branch 'wip-nbalacha-ns-mirroring' of https://github.com/nbalacha/ceph into HEAD 2024-09-12T15:55:58.552 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-12T15:55:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-12T15:55:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-12T15:55:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-12T15:55:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-12T15:55:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-12T15:55:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-12T15:55:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-12T15:55:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-12T15:55:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-12T15:55:59.442 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-09-12T15:55:59.442 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-12T15:55:59.498 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-12T15:55:59.499 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-12T15:55:59.500 DEBUG:teuthology.orchestra.run.smithi019:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d61b08865d4d98738f45ff2e4806d7fb6d03eca1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-12T15:55:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ IMAGE_COUNT=50 2024-09-12T15:55:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ export LOCKDEP=0 2024-09-12T15:55:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ LOCKDEP=0 2024-09-12T15:55:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CLUSTER1=cluster1 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CLUSTER2=cluster2 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ POOL=mirror 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ PARENT_POOL=mirror_parent 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ NS1=ns1 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ NS2=ns2 2024-09-12T15:55:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ TEMPDIR= 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ID=mirror 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ MIRROR_POOL_MODE=image 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' snapshot = snapshot ']' 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ MIRROR_POOL_MODE=image 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS='--id mirror' 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-12T15:55:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-12T15:55:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-12T15:55:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_BIN=. 2024-09-12T15:55:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_SRC=. 2024-09-12T15:55:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-12T15:55:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 -gt 0 ']' 2024-09-12T15:55:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ setup 2024-09-12T15:55:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local c 2024-09-12T15:55:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-12T15:55:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ setup_tempdir 2024-09-12T15:55:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:55:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mktemp -d 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ TEMPDIR=/tmp/tmp.65UL20DnMX 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -z 1 ']' 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ setup_pools cluster1 cluster2 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_cluster=cluster2 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mon_map_file 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mon_addr 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local admin_key_file 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local uuid 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:55:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-12T15:56:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror' already exists 2024-09-12T15:56:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-12T15:56:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror_parent' already exists 2024-09-12T15:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-12T15:56:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-12T15:56:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-12T15:56:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-12T15:56:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-12T15:56:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-12T15:56:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-12T15:56:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-12T15:56:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -z ']' 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -z ']' 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_add cluster1 mirror cluster2 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local client_cluster=cluster2 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_cluster=cluster2 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local uuid_var_name 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local error_code 2024-09-12T15:56:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local peer_uuid 2024-09-12T15:56:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T15:56:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:56:07.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-12T15:56:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_uuid=e4023963-1760-4ed3-98ed-e27160d179a0 2024-09-12T15:56:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T15:56:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 -eq 0 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror_parent 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local client_cluster=cluster2 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_cluster=cluster2 2024-09-12T15:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local uuid_var_name 2024-09-12T15:56:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local error_code 2024-09-12T15:56:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local peer_uuid 2024-09-12T15:56:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T15:56:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:56:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_uuid=b9f61fea-5bd1-44f4-8568-75f594988c9e 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 -eq 0 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ setup_pools cluster2 cluster1 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_cluster=cluster1 2024-09-12T15:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mon_map_file 2024-09-12T15:56:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mon_addr 2024-09-12T15:56:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local admin_key_file 2024-09-12T15:56:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local uuid 2024-09-12T15:56:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-12T15:56:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror' already exists 2024-09-12T15:56:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-12T15:56:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror_parent' already exists 2024-09-12T15:56:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-12T15:56:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-12T15:56:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T15:56:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-12T15:56:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-12T15:56:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-12T15:56:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-12T15:56:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-12T15:56:16.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-12T15:56:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -z ']' 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -z ']' 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_add cluster2 mirror cluster1 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local client_cluster=cluster1 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_cluster=cluster1 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local uuid_var_name 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local error_code 2024-09-12T15:56:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local peer_uuid 2024-09-12T15:56:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T15:56:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:56:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-12T15:56:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_uuid=21aa6e89-b459-42ef-9230-97817db8970d 2024-09-12T15:56:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T15:56:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:56:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 -eq 0 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror_parent 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local client_cluster=cluster1 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_cluster=cluster1 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local uuid_var_name 2024-09-12T15:56:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local error_code 2024-09-12T15:56:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local peer_uuid 2024-09-12T15:56:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 16 32 2024-09-12T15:56:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:56:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ peer_uuid=4c937f77-62dc-4315-94e9-3d1fda7da2e7 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 17 ']' 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 -eq 0 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:56:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:56:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirrors cluster1 2024-09-12T15:56:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 0 3 2024-09-12T15:56:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster1:0 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:0 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster1:0 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster1 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=0 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster1 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=0 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster1:1 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:1 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster1:1 2024-09-12T15:56:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster1 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=1 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster1 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=1 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster1:2 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:2 2024-09-12T15:56:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster1:2 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster1 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=2 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster1 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=2 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T15:56:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster1:3 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:3 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster1:3 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster1 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=3 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster1 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=3 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirrors cluster2 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 0 3 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster2:0 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:0 2024-09-12T15:56:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster2:0 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster2 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=0 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster2 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster2 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=0 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster2:1 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:1 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster2:1 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster2 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=1 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster2 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster2 2024-09-12T15:56:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=1 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster2:2 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:2 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster2:2 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster2 2024-09-12T15:56:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=2 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster2 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster2 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=2 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster2:3 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:3 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-12T15:56:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster2:3 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster2 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=3 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster2 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster2 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=3 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ tee -a /tmp/tmp.65UL20DnMX/rbd-mirror.test.log 2024-09-12T15:56:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ date '+%F %T' 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo 2024-09-12 15:56:16 TEST: add image and test replay after client crashes 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:2024-09-12 15:56:16 TEST: add image and test replay after client crashes 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=test 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T15:56:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror test 512M 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T15:56:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-09-12T15:56:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T15:56:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T15:56:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:56:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-12T15:56:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T15:56:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-12T15:56:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T15:56:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-12T15:56:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T15:56:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T15:56:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T15:56:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T15:56:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a61319f6d9 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a61319f6d9 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:16 2024 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 15:56:16 2024 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:16 2024 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:56:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T15:56:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:56:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:56:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T15:56:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T15:56:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T15:56:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T15:56:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T15:56:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:56:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:56:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:56:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:56:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:56:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:56:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:56:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:56:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:56:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T15:56:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T15:56:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T15:56:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-12T15:56:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T15:56:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T15:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T15:56:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T15:56:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-12T15:56:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 1 2024-09-12T15:56:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:56:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:56:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:56:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:56:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:56:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:56:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:56:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:56:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:56:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:56:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:56:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:56:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:56:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-12T15:56:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T15:56:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T15:56:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T15:56:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-12T15:56:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T15:56:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T15:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T15:56:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-12T15:56:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 2 2024-09-12T15:56:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:56:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:56:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:56:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:56:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:56:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:56:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:56:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:56:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:56:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:56:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:56:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:56:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:56:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:56:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:56:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:56:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:56:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:56:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:56:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T15:56:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T15:56:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T15:56:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T15:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-12T15:56:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-12T15:56:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T15:56:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T15:56:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T15:56:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T15:56:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T15:56:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T15:56:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T15:56:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T15:56:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-12T15:56:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 4 2024-09-12T15:56:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:56:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:56:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:56:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:56:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:56:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:56:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:56:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:56:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:56:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:56:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:56:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:56:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T15:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T15:56:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-12T15:56:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T15:56:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T15:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T15:56:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-12T15:56:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 8 2024-09-12T15:56:34.325 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T15:56:34.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T15:56:34.326 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-12T15:56:34.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T15:56:34.326+0000 7f23e6a38640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 38067) UID: 1000 2024-09-12T15:56:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:56:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:56:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:56:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:56:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:56:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:56:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:56:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:56:34.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-12T15:56:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:56:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:56:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:56:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:56:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:56:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:56:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-12T15:56:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T15:56:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T15:56:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T15:56:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T15:56:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-12T15:56:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-12T15:56:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T15:56:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T15:56:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T15:56:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T15:56:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T15:56:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T15:56:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T15:56:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-12T15:56:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 8 2024-09-12T15:56:41.327 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T15:56:41.327 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T15:56:41.328 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T15:56:41.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T15:56:41.328 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T15:56:41.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T15:56:41.331 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-12T15:56:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:56:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:56:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:56:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:56:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:56:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:56:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:56:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:56:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:56:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:56:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T15:56:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T15:56:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T15:56:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T15:56:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T15:56:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T15:56:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T15:56:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:56:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name= 2024-09-12T15:56:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 10 2024-09-12T15:56:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T15:56:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T15:56:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:56:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T15:56:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T15:56:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:56:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T15:56:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 75139072 bytes 2024-09-12T15:56:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 136766464 bytes 2024-09-12T15:56:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 179460096 bytes 2024-09-12T15:56:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 231145472 bytes 2024-09-12T15:56:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 295505920 bytes 2024-09-12T15:56:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 344574976 bytes 2024-09-12T15:56:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 399096832 bytes 2024-09-12T15:56:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 402236416 bytes 2024-09-12T15:56:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 436830208 bytes 2024-09-12T15:56:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 442650624 bytes 2024-09-12T15:56:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 516672512 bytes 2024-09-12T15:56:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 577741824 bytes 2024-09-12T15:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 644301824 bytes 2024-09-12T15:56:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 713693184 bytes 2024-09-12T15:56:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 761854976 bytes 2024-09-12T15:56:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 783978496 bytes 2024-09-12T15:56:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 797748224 bytes 2024-09-12T15:56:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 840226816 bytes 2024-09-12T15:56:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 908353536 bytes 2024-09-12T15:56:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 961161216 bytes 2024-09-12T15:56:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1065643008 bytes 2024-09-12T15:56:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1085414400 bytes 2024-09-12T15:56:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1094209536 bytes 2024-09-12T15:56:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1109025792 bytes 2024-09-12T15:56:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1163844608 bytes 2024-09-12T15:56:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1187695616 bytes 2024-09-12T15:56:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1224782848 bytes 2024-09-12T15:56:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1249148928 bytes 2024-09-12T15:56:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1266567168 bytes 2024-09-12T15:56:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1325468672 bytes 2024-09-12T15:56:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1430090752 bytes 2024-09-12T15:56:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1504518144 bytes 2024-09-12T15:56:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1532327936 bytes 2024-09-12T15:56:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1543121920 bytes 2024-09-12T15:56:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1583372288 bytes 2024-09-12T15:56:49.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29715759071613623) > weight (0.25) 2024-09-12T15:56:49.325 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-12T15:56:49.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-12T15:56:49.326 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-12T15:56:49.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T15:56:49.325+0000 7f5a2518c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38148) UID: 1000 2024-09-12T15:56:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1631196160 bytes 2024-09-12T15:56:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1732858880 bytes 2024-09-12T15:56:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1764850688 bytes 2024-09-12T15:56:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1787899904 bytes 2024-09-12T15:56:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1808174080 bytes 2024-09-12T15:56:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1848011776 bytes 2024-09-12T15:56:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1930159104 bytes 2024-09-12T15:56:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1985788928 bytes 2024-09-12T15:56:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2046186496 bytes 2024-09-12T15:56:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2079599616 bytes 2024-09-12T15:56:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2104561664 bytes 2024-09-12T15:56:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2163784704 bytes 2024-09-12T15:56:51.326 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-12T15:56:51.327 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T15:56:51.327 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-12T15:56:51.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-12T15:56:51.327 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-12T15:56:51.330 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-12T15:56:51.330 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-12T15:56:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2215175168 bytes 2024-09-12T15:56:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2290291712 bytes 2024-09-12T15:56:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2317215744 bytes 2024-09-12T15:56:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2386202624 bytes 2024-09-12T15:56:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2386812928 bytes 2024-09-12T15:56:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2467275776 bytes 2024-09-12T15:56:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2479969280 bytes 2024-09-12T15:56:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2527904768 bytes 2024-09-12T15:56:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2572765184 bytes 2024-09-12T15:56:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2615779328 bytes 2024-09-12T15:56:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2640338944 bytes 2024-09-12T15:56:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2645390336 bytes 2024-09-12T15:56:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2666085376 bytes 2024-09-12T15:56:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2705785856 bytes 2024-09-12T15:56:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2719970304 bytes 2024-09-12T15:56:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2759503872 bytes 2024-09-12T15:56:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2794324992 bytes 2024-09-12T15:56:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2818420736 bytes 2024-09-12T15:56:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2832684032 bytes 2024-09-12T15:56:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2873170944 bytes 2024-09-12T15:56:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2880950272 bytes 2024-09-12T15:56:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T15:56:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:56:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T15:56:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:56:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:56:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T15:56:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:56:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:56:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:56:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:56:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:56:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: down+unknown 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: status not found 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T15:56:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: down+unknown 2024-09-12T15:56:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: status not found 2024-09-12T15:56:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: ' 2024-09-12T15:56:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T15:56:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 1 2024-09-12T15:56:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 2 2024-09-12T15:56:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T15:56:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:56:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:56:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:56:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:56:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:56:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:56:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:56:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:56:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T15:56:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T15:56:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:56:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: down+unknown 2024-09-12T15:56:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: status not found 2024-09-12T15:56:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12T15:56:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T15:56:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T15:56:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: down+unknown 2024-09-12T15:56:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: status not found 2024-09-12T15:56:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: ' 2024-09-12T15:56:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T15:56:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 1 2024-09-12T15:56:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:56:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 4 2024-09-12T15:57:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:57:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T15:57:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T15:57:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:57:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: down+unknown 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: status not found 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: down+unknown 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: status not found 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: ' 2024-09-12T15:57:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T15:57:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 1 2024-09-12T15:57:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:57:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 8 2024-09-12T15:57:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:57:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T15:57:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T15:57:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:57:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T15:57:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156577,"remote_snapshot_timestamp":1726156577,"replay_state":"idle"} 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:57:06 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:57:06' 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T15:57:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T15:57:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T15:57:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156577,"remote_snapshot_timestamp":1726156577,"replay_state":"idle"} 2024-09-12T15:57:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:57:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:57:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T15:57:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:57:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T15:57:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T15:57:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T15:57:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T15:57:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:57:06' 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 15:57:06' 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 15:57:06' 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:57:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T15:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK42224267mirror.3smithi01919.3.0-4883-gd61b0886falseOK43034306mirror.0smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156577,"remote_snapshot_timestamp":1726156577,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 15:57:06cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 15:57:06' 2024-09-12T15:57:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T15:57:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T15:57:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:57:06' 2024-09-12T15:57:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T15:57:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T15:57:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:57:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T15:57:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:57:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap1 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap1 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap1 2024-09-12T15:57:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-12T15:57:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:57:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:57:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T15:57:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T15:57:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T15:57:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T15:57:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T15:57:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:57:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:57:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:57:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:57:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:57:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:57:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:57:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:57:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:57:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:57:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:57:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:57:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:57:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:57:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:57:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:57:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:57:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:57:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:57:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:57:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:57:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:57:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:57:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:57:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T15:57:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T15:57:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T15:57:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T15:57:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T15:57:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:57:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:57:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:57:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:57:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T15:57:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T15:57:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:57:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:57:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T15:57:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T15:57:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:57:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T15:57:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T15:57:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 5 2024-09-12T15:57:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:57:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:57:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T15:57:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:57:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T15:57:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=5 2024-09-12T15:57:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:57:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:57:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T15:57:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:57:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:57:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:57:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=3 2024-09-12T15:57:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 5 = 3 2024-09-12T15:57:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:57:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T15:57:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:57:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:57:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:57:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:57:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:57:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:57:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:57:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:57:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 5 = '' 2024-09-12T15:57:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:57:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T15:57:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:57:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:57:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:57:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:57:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:57:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:57:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:57:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:57:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 5 = '' 2024-09-12T15:57:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:57:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T15:57:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:57:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:57:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=5 2024-09-12T15:57:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 5 = 5 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap1 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T15:57:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:57:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T15:57:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:57:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:57:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:57:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T15:57:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T15:57:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T15:57:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:57:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap1 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap1 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:57:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:57:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:57:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:57:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:57:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:57:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:57:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:57:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:57:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap1-clean 2024-09-12T15:57:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-12T15:57:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap1-clean 2024-09-12T15:57:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T15:57:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T15:57:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T15:57:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T15:57:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=10 2024-09-12T15:57:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:57:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T15:57:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 60475392 bytes 2024-09-12T15:57:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 110453760 bytes 2024-09-12T15:57:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 170368000 bytes 2024-09-12T15:57:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 202500096 bytes 2024-09-12T15:57:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 226598912 bytes 2024-09-12T15:57:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 328413184 bytes 2024-09-12T15:57:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 401676288 bytes 2024-09-12T15:57:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 444070912 bytes 2024-09-12T15:57:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 540672000 bytes 2024-09-12T15:57:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 566345728 bytes 2024-09-12T15:57:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 608459776 bytes 2024-09-12T15:57:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 637577216 bytes 2024-09-12T15:57:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 662708224 bytes 2024-09-12T15:57:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 731045888 bytes 2024-09-12T15:57:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 784699392 bytes 2024-09-12T15:57:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 851007488 bytes 2024-09-12T15:57:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 915914752 bytes 2024-09-12T15:57:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 926766080 bytes 2024-09-12T15:57:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 955759616 bytes 2024-09-12T15:57:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 978270208 bytes 2024-09-12T15:57:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 989919232 bytes 2024-09-12T15:57:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1022330880 bytes 2024-09-12T15:57:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1058607104 bytes 2024-09-12T15:57:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1119300608 bytes 2024-09-12T15:57:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1199212544 bytes 2024-09-12T15:57:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1240502272 bytes 2024-09-12T15:57:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1288819712 bytes 2024-09-12T15:57:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1326534656 bytes 2024-09-12T15:57:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1376749568 bytes 2024-09-12T15:57:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1442388992 bytes 2024-09-12T15:57:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1475954688 bytes 2024-09-12T15:57:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1491066880 bytes 2024-09-12T15:57:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1552527360 bytes 2024-09-12T15:57:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1555734528 bytes 2024-09-12T15:57:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1635123200 bytes 2024-09-12T15:57:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1705655296 bytes 2024-09-12T15:57:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1727399936 bytes 2024-09-12T15:57:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1783367680 bytes 2024-09-12T15:57:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1785654272 bytes 2024-09-12T15:57:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1811511296 bytes 2024-09-12T15:57:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1844276224 bytes 2024-09-12T15:57:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1905990656 bytes 2024-09-12T15:57:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1935714304 bytes 2024-09-12T15:57:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2046504960 bytes 2024-09-12T15:57:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2065521664 bytes 2024-09-12T15:57:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2107196416 bytes 2024-09-12T15:57:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2113779712 bytes 2024-09-12T15:57:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2168480768 bytes 2024-09-12T15:57:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2213443584 bytes 2024-09-12T15:57:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2261181440 bytes 2024-09-12T15:57:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2285438976 bytes 2024-09-12T15:57:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2324849664 bytes 2024-09-12T15:57:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2382663680 bytes 2024-09-12T15:57:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2386663424 bytes 2024-09-12T15:57:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2409497600 bytes 2024-09-12T15:57:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2416276480 bytes 2024-09-12T15:57:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2465336320 bytes 2024-09-12T15:57:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2541913088 bytes 2024-09-12T15:57:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2561387520 bytes 2024-09-12T15:57:43.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2562021376 bytes 2024-09-12T15:57:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2584551424 bytes 2024-09-12T15:57:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2628507648 bytes 2024-09-12T15:57:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2647868416 bytes 2024-09-12T15:57:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2695569408 bytes 2024-09-12T15:57:43.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2714678272 bytes 2024-09-12T15:57:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T15:57:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:57:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T15:57:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T15:57:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:57:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:57:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:57:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:57:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:57:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T15:57:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:57:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:57:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:57:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:57:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:57:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T15:57:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T15:57:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:57:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":17565969.07,"bytes_per_snapshot":263489536.0,"last_snapshot_bytes":526979072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156647,"remote_snapshot_timestamp":1726156647,"replay_state":"idle"} 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:57:36 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:57:36' 2024-09-12T15:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T15:57:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T15:57:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T15:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":17565969.07,"bytes_per_snapshot":263489536.0,"last_snapshot_bytes":526979072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156647,"remote_snapshot_timestamp":1726156647,"replay_state":"idle"} 2024-09-12T15:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T15:57:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:57:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T15:57:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T15:57:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T15:57:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:57:36' 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 15:57:36' 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 15:57:36' 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:57:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:57:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T15:57:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK42224267mirror.3smithi01919.3.0-4883-gd61b0886falseOK43034306mirror.0smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":17565969.07,"bytes_per_snapshot":263489536.0,"last_snapshot_bytes":526979072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156647,"remote_snapshot_timestamp":1726156647,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 15:57:36cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 15:57:36' 2024-09-12T15:57:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T15:57:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T15:57:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:57:36' 2024-09-12T15:57:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T15:57:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T15:57:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:57:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T15:57:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap2 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap2 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap2 2024-09-12T15:57:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-12T15:58:03.332 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9769651439967649) > weight (0.25) 2024-09-12T15:58:03.332 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.36203606414671097) > weight (0.25) 2024-09-12T15:58:03.332 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8683422707743974) > weight (0.25) 2024-09-12T15:58:03.332 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-12T15:58:03.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-12T15:58:03.334 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-12T15:58:03.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T15:58:03.333+0000 7fd1b4ed7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 38115) UID: 1000 2024-09-12T15:58:11.335 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-12T15:58:11.335 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T15:58:11.335 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-12T15:58:11.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-12T15:58:11.336 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-12T15:58:11.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-12T15:58:11.340 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-12T15:58:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T15:58:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T15:58:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T15:58:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:58:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:58:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T15:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T15:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T15:58:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T15:58:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T15:58:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T15:58:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:58:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T15:58:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T15:58:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T15:58:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T15:58:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:58:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T15:58:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:58:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:58:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:58:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:58:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:58:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:58:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:58:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:58:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:58:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:58:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:58:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:58:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:58:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:58:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:58:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:58:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:58:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:58:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:58:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:58:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:58:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:58:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:58:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:58:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:58:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:58:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:58:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:58:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:58:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:58:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:58:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:58:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:58:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:58:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:58:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:58:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:58:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:58:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:58:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:58:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:58:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:58:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:58:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:58:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:58:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:58:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T15:58:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T15:58:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T15:58:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T15:58:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T15:58:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T15:58:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T15:58:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:58:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T15:58:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T15:58:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T15:58:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 8 2024-09-12T15:58:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:58:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:58:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:58:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T15:58:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:58:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T15:58:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=8 2024-09-12T15:58:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:58:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:58:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T15:58:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:58:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:58:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:58:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 8 = '' 2024-09-12T15:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T15:58:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:58:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:58:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:58:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:58:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:58:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:58:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:58:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:58:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 8 = '' 2024-09-12T15:58:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:58:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T15:58:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:58:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:58:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:58:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:58:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:58:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:58:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:58:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:58:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 8 = '' 2024-09-12T15:58:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:58:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T15:58:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:58:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:58:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:58:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:58:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=8 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 8 = 8 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap2 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T15:58:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:58:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-12T15:58:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T15:58:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T15:58:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T15:58:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 4 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T15:58:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T15:58:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T15:58:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:58:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap1-clean ']' 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap1-clean 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:58:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:58:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:58:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap2 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap2 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:58:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:58:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:58:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap2-clean 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap2-clean 2024-09-12T15:58:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T15:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T15:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T15:58:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:58:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T15:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T15:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T15:58:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 20005888 bytes 2024-09-12T15:58:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 52785152 bytes 2024-09-12T15:58:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 87903232 bytes 2024-09-12T15:58:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 154237952 bytes 2024-09-12T15:58:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 161719296 bytes 2024-09-12T15:58:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 222873600 bytes 2024-09-12T15:58:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 270879744 bytes 2024-09-12T15:58:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 271514624 bytes 2024-09-12T15:58:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 320643072 bytes 2024-09-12T15:58:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 345957376 bytes 2024-09-12T15:58:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 428264448 bytes 2024-09-12T15:58:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 470226944 bytes 2024-09-12T15:58:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 493629440 bytes 2024-09-12T15:58:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 537847808 bytes 2024-09-12T15:58:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 538733568 bytes 2024-09-12T15:58:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 577568768 bytes 2024-09-12T15:58:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 628655104 bytes 2024-09-12T15:58:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 638482432 bytes 2024-09-12T15:58:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 741609472 bytes 2024-09-12T15:58:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 761337856 bytes 2024-09-12T15:58:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 777287680 bytes 2024-09-12T15:58:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 795641856 bytes 2024-09-12T15:58:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 810467328 bytes 2024-09-12T15:58:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 813036544 bytes 2024-09-12T15:58:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 843357184 bytes 2024-09-12T15:58:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 843990016 bytes 2024-09-12T15:58:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 911039488 bytes 2024-09-12T15:58:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 935668736 bytes 2024-09-12T15:58:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 976999424 bytes 2024-09-12T15:58:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 994454528 bytes 2024-09-12T15:58:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1029571584 bytes 2024-09-12T15:58:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1092774912 bytes 2024-09-12T15:58:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1132396544 bytes 2024-09-12T15:58:29.928 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T15:58:29.926+0000 7fbc5a793640 -1 reset not still connected to 0x55f81cbe2750 2024-09-12T15:58:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1197514752 bytes 2024-09-12T15:58:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1217702912 bytes 2024-09-12T15:58:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1307971584 bytes 2024-09-12T15:58:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1402306560 bytes 2024-09-12T15:58:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1405151232 bytes 2024-09-12T15:58:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1452308480 bytes 2024-09-12T15:58:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1475477504 bytes 2024-09-12T15:58:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1535435776 bytes 2024-09-12T15:58:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1580076032 bytes 2024-09-12T15:58:31.331 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-12T15:58:31.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-12T15:58:31.332 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-12T15:58:31.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T15:58:31.330+0000 7f95e4c73640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 38134) UID: 1000 2024-09-12T15:58:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1625638912 bytes 2024-09-12T15:58:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1639496704 bytes 2024-09-12T15:58:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1714338816 bytes 2024-09-12T15:58:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1795166208 bytes 2024-09-12T15:58:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1839673344 bytes 2024-09-12T15:58:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1866743808 bytes 2024-09-12T15:58:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1869045760 bytes 2024-09-12T15:58:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1889015808 bytes 2024-09-12T15:58:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 1928313856 bytes 2024-09-12T15:58:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 1941043200 bytes 2024-09-12T15:58:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 1991742464 bytes 2024-09-12T15:58:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2082399232 bytes 2024-09-12T15:58:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2165052416 bytes 2024-09-12T15:58:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2222553088 bytes 2024-09-12T15:58:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2223153152 bytes 2024-09-12T15:58:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2250472448 bytes 2024-09-12T15:58:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2262026240 bytes 2024-09-12T15:58:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2308217856 bytes 2024-09-12T15:58:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2359053312 bytes 2024-09-12T15:58:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2376721408 bytes 2024-09-12T15:58:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T15:58:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:58:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T15:58:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T15:58:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:58:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:58:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:58:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T15:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T15:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T15:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":17565969.07,"bytes_per_snapshot":263489536.0,"last_snapshot_bytes":526979072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156647,"remote_snapshot_timestamp":1726156647,"replay_state":"idle"} 2024-09-12T15:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T15:58:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:58:06 2024-09-12T15:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T15:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T15:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: down+stopped 2024-09-12T15:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T15:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:58:06' 2024-09-12T15:58:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T15:58:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T15:58:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T15:58:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":17565969.07,"bytes_per_snapshot":263489536.0,"last_snapshot_bytes":526979072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156647,"remote_snapshot_timestamp":1726156647,"replay_state":"idle"} 2024-09-12T15:58:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:58:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:58:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T15:58:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:58:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T15:58:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T15:58:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T15:58:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T15:58:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:58:06' 2024-09-12T15:58:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 15:58:06' 2024-09-12T15:58:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:58:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 15:58:06' 2024-09-12T15:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:58:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T15:58:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='WARNINGOKWARNING142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43034306mirror.0smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":17565969.07,"bytes_per_snapshot":263489536.0,"last_snapshot_bytes":526979072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156647,"remote_snapshot_timestamp":1726156647,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 15:58:06cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8down+stoppedlocal image is primary2024-09-12 15:58:06' 2024-09-12T15:58:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T15:58:35.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T15:58:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:58:06' 2024-09-12T15:58:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T15:58:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T15:58:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:58:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T15:58:35.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap3 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap3 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:58:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap3 2024-09-12T15:58:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-12T15:58:40.333 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-12T15:58:40.334 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T15:58:40.334 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-12T15:58:40.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-12T15:58:40.334 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-12T15:58:40.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-12T15:58:40.341 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-12T15:59:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T15:59:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T15:59:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T15:59:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:59:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:59:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:59:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T15:59:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T15:59:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T15:59:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T15:59:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T15:59:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T15:59:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:59:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T15:59:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T15:59:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T15:59:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T15:59:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:59:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T15:59:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:59:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:59:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:59:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:59:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:59:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:59:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:59:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:59:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:59:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:59:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:59:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:59:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:59:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:59:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:59:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:59:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:59:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:59:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:59:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:59:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:59:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:59:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:59:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:59:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:59:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:59:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:59:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:59:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:59:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:59:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:59:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:59:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:59:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:59:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:59:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:59:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:59:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:59:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:59:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:59:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:59:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:59:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:59:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:59:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T15:59:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T15:59:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T15:59:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T15:59:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T15:59:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T15:59:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T15:59:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T15:59:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:59:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T15:59:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T15:59:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 11 2024-09-12T15:59:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:59:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:59:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T15:59:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:59:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T15:59:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=11 2024-09-12T15:59:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:59:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:59:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T15:59:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:59:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:59:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:59:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:59:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:59:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:59:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:59:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:59:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 11 = '' 2024-09-12T15:59:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:59:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T15:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:59:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:59:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:59:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:59:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:59:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:59:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:59:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 11 = '' 2024-09-12T15:59:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:59:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T15:59:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:59:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:59:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T15:59:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:59:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T15:59:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T15:59:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T15:59:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T15:59:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 11 = '' 2024-09-12T15:59:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T15:59:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T15:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T15:59:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T15:59:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T15:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=11 2024-09-12T15:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 11 = 11 2024-09-12T15:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-12T15:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap3 2024-09-12T15:59:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:59:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T15:59:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:59:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-12T15:59:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 6 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T15:59:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 15:58:22 2024 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:59:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T15:59:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap2-clean ']' 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap2-clean 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:59:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:59:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:59:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap3 2024-09-12T15:59:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap3 2024-09-12T15:59:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T15:59:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:59:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T15:59:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:59:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T15:59:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T15:59:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap3-clean 2024-09-12T15:59:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-12T15:59:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap3-clean 2024-09-12T15:59:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T15:59:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T15:59:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T15:59:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T15:59:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T15:59:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T15:59:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T15:59:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 19897344 bytes 2024-09-12T15:59:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 36907008 bytes 2024-09-12T15:59:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 69265408 bytes 2024-09-12T15:59:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 98342912 bytes 2024-09-12T15:59:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 138043392 bytes 2024-09-12T15:59:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 167910400 bytes 2024-09-12T15:59:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 248050688 bytes 2024-09-12T15:59:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 307067904 bytes 2024-09-12T15:59:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 379641856 bytes 2024-09-12T15:59:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 430989312 bytes 2024-09-12T15:59:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 502134784 bytes 2024-09-12T15:59:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 536108032 bytes 2024-09-12T15:59:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 593459200 bytes 2024-09-12T15:59:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 685518848 bytes 2024-09-12T15:59:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 726709248 bytes 2024-09-12T15:59:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 815223808 bytes 2024-09-12T15:59:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 861549568 bytes 2024-09-12T15:59:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 908346368 bytes 2024-09-12T15:59:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 960179200 bytes 2024-09-12T15:59:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1015001088 bytes 2024-09-12T15:59:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1015583744 bytes 2024-09-12T15:59:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1055699968 bytes 2024-09-12T15:59:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1127573504 bytes 2024-09-12T15:59:18.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1141356544 bytes 2024-09-12T15:59:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1161217024 bytes 2024-09-12T15:59:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1247188992 bytes 2024-09-12T15:59:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1301109760 bytes 2024-09-12T15:59:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1417567232 bytes 2024-09-12T15:59:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1424507904 bytes 2024-09-12T15:59:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1444737024 bytes 2024-09-12T15:59:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1497114624 bytes 2024-09-12T15:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1606151168 bytes 2024-09-12T15:59:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1690536960 bytes 2024-09-12T15:59:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1716977664 bytes 2024-09-12T15:59:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1722492928 bytes 2024-09-12T15:59:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1784625152 bytes 2024-09-12T15:59:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1858903040 bytes 2024-09-12T15:59:20.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1917206528 bytes 2024-09-12T15:59:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1992115200 bytes 2024-09-12T15:59:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 2069265408 bytes 2024-09-12T15:59:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 2097982464 bytes 2024-09-12T15:59:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 2102257664 bytes 2024-09-12T15:59:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2122236928 bytes 2024-09-12T15:59:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2167291904 bytes 2024-09-12T15:59:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2222229504 bytes 2024-09-12T15:59:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2312281088 bytes 2024-09-12T15:59:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2415910912 bytes 2024-09-12T15:59:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2485819392 bytes 2024-09-12T15:59:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2534564864 bytes 2024-09-12T15:59:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2545876992 bytes 2024-09-12T15:59:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2569469952 bytes 2024-09-12T15:59:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2617168896 bytes 2024-09-12T15:59:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2653065216 bytes 2024-09-12T15:59:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2701471744 bytes 2024-09-12T15:59:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2736221184 bytes 2024-09-12T15:59:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2757870592 bytes 2024-09-12T15:59:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2774189056 bytes 2024-09-12T15:59:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2819100672 bytes 2024-09-12T15:59:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2858115072 bytes 2024-09-12T15:59:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2892499968 bytes 2024-09-12T15:59:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2936651776 bytes 2024-09-12T15:59:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2966799360 bytes 2024-09-12T15:59:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2979088384 bytes 2024-09-12T15:59:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2993888256 bytes 2024-09-12T15:59:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2995708928 bytes 2024-09-12T15:59:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 3064770560 bytes 2024-09-12T15:59:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 3086860288 bytes 2024-09-12T15:59:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 3171002368 bytes 2024-09-12T15:59:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 3207203840 bytes 2024-09-12T15:59:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 3245363200 bytes 2024-09-12T15:59:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 3285181440 bytes 2024-09-12T15:59:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3327142912 bytes 2024-09-12T15:59:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 3361710080 bytes 2024-09-12T15:59:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 3378625536 bytes 2024-09-12T15:59:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T15:59:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T15:59:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T15:59:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:59:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:59:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T15:59:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:59:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T15:59:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T15:59:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:59:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:59:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T15:59:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T15:59:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T15:59:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T15:59:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531457536.0,"last_snapshot_bytes":535936000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156696,"remote_snapshot_timestamp":1726156696,"replay_state":"idle"} 2024-09-12T15:59:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T15:59:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:59:06 2024-09-12T15:59:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T15:59:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T15:59:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T15:59:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T15:59:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:59:21' 2024-09-12T15:59:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T15:59:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T15:59:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T15:59:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531457536.0,"last_snapshot_bytes":535936000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156696,"remote_snapshot_timestamp":1726156696,"replay_state":"idle"} 2024-09-12T15:59:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T15:59:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:59:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T15:59:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:59:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T15:59:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T15:59:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T15:59:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T15:59:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:59:06' 2024-09-12T15:59:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 15:59:06' 2024-09-12T15:59:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T15:59:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 15:59:06' 2024-09-12T15:59:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T15:59:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T15:59:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T15:59:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43034306mirror.0smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531457536.0,"last_snapshot_bytes":535936000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156696,"remote_snapshot_timestamp":1726156696,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 15:59:06cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 15:59:21' 2024-09-12T15:59:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T15:59:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T15:59:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 15:59:06' 2024-09-12T15:59:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T15:59:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T15:59:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T15:59:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T15:59:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T15:59:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap4 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap4 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap4 2024-09-12T15:59:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-12T15:59:33.340 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T15:59:33.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T15:59:33.341 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-12T15:59:33.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T15:59:33.341+0000 7fd8b7fb6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39652) UID: 1000 2024-09-12T15:59:36.342 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T15:59:36.342 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T15:59:36.342 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T15:59:36.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T15:59:36.343 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T15:59:36.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T15:59:36.346 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-12T15:59:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:59:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T15:59:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T15:59:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T15:59:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T15:59:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T15:59:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T15:59:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T15:59:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T15:59:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T15:59:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:59:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T15:59:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T15:59:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T15:59:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T15:59:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T15:59:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T15:59:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T15:59:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:59:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:59:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T15:59:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:59:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T15:59:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T15:59:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:59:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:59:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T15:59:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T15:59:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:59:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T15:59:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T15:59:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:59:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:59:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T15:59:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T15:59:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T15:59:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T15:59:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T15:59:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T15:59:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T15:59:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T15:59:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T15:59:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T15:59:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T15:59:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T15:59:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T15:59:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T15:59:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:59:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:59:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T15:59:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T15:59:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T15:59:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T15:59:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T15:59:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T15:59:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T15:59:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T15:59:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T15:59:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T15:59:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:59:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:59:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:59:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:59:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:59:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T15:59:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T15:59:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:00:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 14 2024-09-12T16:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:00:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:00:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:00:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=14 2024-09-12T16:00:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:00:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:00:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:00:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:00:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:00:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 14 = '' 2024-09-12T16:00:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:00:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:00:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:00:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:00:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:00:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 14 = '' 2024-09-12T16:00:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:00:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:00:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:00:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:00:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:00:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 14 = '' 2024-09-12T16:00:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:00:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:03.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=14 2024-09-12T16:00:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 14 = 14 2024-09-12T16:00:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-12T16:00:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap4 2024-09-12T16:00:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:00:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:00:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:00:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-12T16:00:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:00:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:00:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 8 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 15:58:22 2024 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:00:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:00:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:00:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:00:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap3-clean ']' 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap3-clean 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:00:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:00:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:00:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap4 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap4 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:00:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:00:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:00:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap4-clean 2024-09-12T16:00:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-12T16:00:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap4-clean 2024-09-12T16:00:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T16:00:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:00:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T16:00:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:00:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T16:00:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:00:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T16:00:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 1428480 bytes 2024-09-12T16:00:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 49566720 bytes 2024-09-12T16:00:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 50198528 bytes 2024-09-12T16:00:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 65902592 bytes 2024-09-12T16:00:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 129602560 bytes 2024-09-12T16:00:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 167537664 bytes 2024-09-12T16:00:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 187961344 bytes 2024-09-12T16:00:09.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 224918528 bytes 2024-09-12T16:00:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 246364160 bytes 2024-09-12T16:00:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 265955328 bytes 2024-09-12T16:00:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 305585152 bytes 2024-09-12T16:00:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 374255616 bytes 2024-09-12T16:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 404385792 bytes 2024-09-12T16:00:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 445547520 bytes 2024-09-12T16:00:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 463417344 bytes 2024-09-12T16:00:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 493478912 bytes 2024-09-12T16:00:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 525740032 bytes 2024-09-12T16:00:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 531832832 bytes 2024-09-12T16:00:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 547691520 bytes 2024-09-12T16:00:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 579480576 bytes 2024-09-12T16:00:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 756808704 bytes 2024-09-12T16:00:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 829671424 bytes 2024-09-12T16:00:11.342 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9326111102422143) > weight (0.25) 2024-09-12T16:00:11.342 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.28328247082214675) > weight (0.25) 2024-09-12T16:00:11.342 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-12T16:00:11.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-12T16:00:11.343 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-12T16:00:11.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:00:11.347+0000 7f193bdd3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 38166) UID: 1000 2024-09-12T16:00:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 862589952 bytes 2024-09-12T16:00:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 905371648 bytes 2024-09-12T16:00:11.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 950227968 bytes 2024-09-12T16:00:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1038073856 bytes 2024-09-12T16:00:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1051385856 bytes 2024-09-12T16:00:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1123362816 bytes 2024-09-12T16:00:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1168826368 bytes 2024-09-12T16:00:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1189398528 bytes 2024-09-12T16:00:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1257394176 bytes 2024-09-12T16:00:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1317173248 bytes 2024-09-12T16:00:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1374295040 bytes 2024-09-12T16:00:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1429181440 bytes 2024-09-12T16:00:13.343 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-12T16:00:13.343 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:00:13.344 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-12T16:00:13.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-12T16:00:13.344 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-12T16:00:13.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-12T16:00:13.350 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-12T16:00:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1468760064 bytes 2024-09-12T16:00:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1537429504 bytes 2024-09-12T16:00:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1541002240 bytes 2024-09-12T16:00:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1541622784 bytes 2024-09-12T16:00:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1569169408 bytes 2024-09-12T16:00:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1578485760 bytes 2024-09-12T16:00:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1613202432 bytes 2024-09-12T16:00:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1667248128 bytes 2024-09-12T16:00:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1688474624 bytes 2024-09-12T16:00:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1743755264 bytes 2024-09-12T16:00:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1759766528 bytes 2024-09-12T16:00:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1813049344 bytes 2024-09-12T16:00:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1886384128 bytes 2024-09-12T16:00:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1929534464 bytes 2024-09-12T16:00:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1952959488 bytes 2024-09-12T16:00:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2026162176 bytes 2024-09-12T16:00:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2091047936 bytes 2024-09-12T16:00:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2151750656 bytes 2024-09-12T16:00:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2264250368 bytes 2024-09-12T16:00:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2332716032 bytes 2024-09-12T16:00:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2344274944 bytes 2024-09-12T16:00:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2366130176 bytes 2024-09-12T16:00:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2381015040 bytes 2024-09-12T16:00:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2442993664 bytes 2024-09-12T16:00:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2487300096 bytes 2024-09-12T16:00:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2502134784 bytes 2024-09-12T16:00:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2540288000 bytes 2024-09-12T16:00:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2562420736 bytes 2024-09-12T16:00:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2605771776 bytes 2024-09-12T16:00:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2653237248 bytes 2024-09-12T16:00:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2672869376 bytes 2024-09-12T16:00:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2680480768 bytes 2024-09-12T16:00:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2701197312 bytes 2024-09-12T16:00:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2758947840 bytes 2024-09-12T16:00:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2824662016 bytes 2024-09-12T16:00:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2836129792 bytes 2024-09-12T16:00:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 2845636608 bytes 2024-09-12T16:00:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 2878437376 bytes 2024-09-12T16:00:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 2911589376 bytes 2024-09-12T16:00:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 2916135936 bytes 2024-09-12T16:00:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 2938277888 bytes 2024-09-12T16:00:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7600 IOs, 2968348672 bytes 2024-09-12T16:00:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7700 IOs, 3013796864 bytes 2024-09-12T16:00:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7800 IOs, 3052992512 bytes 2024-09-12T16:00:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7900 IOs, 3136473088 bytes 2024-09-12T16:00:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:00:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:00:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:00:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T16:00:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T16:00:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":17895253.33,"bytes_per_snapshot":534576640.0,"last_snapshot_bytes":536857600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726156799,"remote_snapshot_timestamp":1726156799,"replay_state":"idle"} 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:00:06 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 15:59:51' 2024-09-12T16:00:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T16:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T16:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T16:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":17895253.33,"bytes_per_snapshot":534576640.0,"last_snapshot_bytes":536857600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726156799,"remote_snapshot_timestamp":1726156799,"replay_state":"idle"} 2024-09-12T16:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T16:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T16:00:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T16:00:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T16:00:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:00:06' 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 16:00:06' 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 16:00:06' 2024-09-12T16:00:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:00:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:00:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T16:00:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOK44284431mirror.0smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":17895253.33,"bytes_per_snapshot":534576640.0,"last_snapshot_bytes":536857600,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726156799,"remote_snapshot_timestamp":1726156799,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 16:00:06cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 15:59:51' 2024-09-12T16:00:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T16:00:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T16:00:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:00:06' 2024-09-12T16:00:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T16:00:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T16:00:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:00:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T16:00:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap5 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap5 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap5 2024-09-12T16:00:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:00:36.412 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:00:36.410+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b121c000 2024-09-12T16:00:36.413 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:00:36.412+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b121d930 2024-09-12T16:00:36.414 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:00:36.412+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b121de10 2024-09-12T16:00:43.346 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8975132497759044) > weight (0.25) 2024-09-12T16:00:43.347 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6775590766361218) > weight (0.25) 2024-09-12T16:00:43.347 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6089624846561323) > weight (0.25) 2024-09-12T16:00:43.347 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5045785185840749) > weight (0.25) 2024-09-12T16:00:43.347 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-12T16:00:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:00:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:00:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T16:00:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:00:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:00:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:00:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:00:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:00:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:00:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:00:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T16:00:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:00:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:00:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:00:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:00:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:00:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:00:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:00:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:00:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:00:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:00:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:00:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:00:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:00:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T16:00:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T16:00:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:00:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:00:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:00:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T16:00:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:00:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:00:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:00:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:00:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:00:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:00:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:00:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:00:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:00:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:00:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T16:00:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:00:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:00:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:00:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:00:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:00:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:00:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:00:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:00:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:00:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T16:00:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T16:00:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:00:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:00:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T16:00:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T16:00:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:00:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T16:00:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:00:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 17 2024-09-12T16:00:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:00:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:00:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:00:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:00:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:00:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=17 2024-09-12T16:00:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:00:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:00:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:00:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:00:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:00:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 17 = '' 2024-09-12T16:00:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:00:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:00:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:00:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:00:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:00:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 17 = '' 2024-09-12T16:00:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:00:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:00:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:00:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:00:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:00:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:00:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 17 = '' 2024-09-12T16:00:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:00:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:00:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:00:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:00:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:00:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:00:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=17 2024-09-12T16:00:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 17 = 17 2024-09-12T16:00:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-12T16:00:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:00:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap5 2024-09-12T16:00:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:00:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:00:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:00:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-12T16:00:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 10 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:00:05 2024 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:00:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:00:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:00:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:00:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:00:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap4-clean ']' 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap4-clean 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:00:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:00:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:00:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:00:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:00.004 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap5 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap5 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:01:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:01:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:01:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap5-clean 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap5-clean 2024-09-12T16:01:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T16:01:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:01:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T16:01:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:01:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:01:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:01:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T16:01:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 30268416 bytes 2024-09-12T16:01:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 72886272 bytes 2024-09-12T16:01:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 177485824 bytes 2024-09-12T16:01:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 247662592 bytes 2024-09-12T16:01:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 321937408 bytes 2024-09-12T16:01:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 356706304 bytes 2024-09-12T16:01:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 422711296 bytes 2024-09-12T16:01:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 469152768 bytes 2024-09-12T16:01:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 496130048 bytes 2024-09-12T16:01:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 521939968 bytes 2024-09-12T16:01:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 546217984 bytes 2024-09-12T16:01:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 608014336 bytes 2024-09-12T16:01:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 651604992 bytes 2024-09-12T16:01:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 714988544 bytes 2024-09-12T16:01:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 716129280 bytes 2024-09-12T16:01:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 735502336 bytes 2024-09-12T16:01:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 758362112 bytes 2024-09-12T16:01:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 821645312 bytes 2024-09-12T16:01:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 848114688 bytes 2024-09-12T16:01:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 876782592 bytes 2024-09-12T16:01:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 894409728 bytes 2024-09-12T16:01:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 947355648 bytes 2024-09-12T16:01:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1015691264 bytes 2024-09-12T16:01:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1073657856 bytes 2024-09-12T16:01:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1112055808 bytes 2024-09-12T16:01:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1171605504 bytes 2024-09-12T16:01:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1191156736 bytes 2024-09-12T16:01:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1218369536 bytes 2024-09-12T16:01:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1270305792 bytes 2024-09-12T16:01:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1384015872 bytes 2024-09-12T16:01:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1412306944 bytes 2024-09-12T16:01:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1451635712 bytes 2024-09-12T16:01:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1512863744 bytes 2024-09-12T16:01:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1544878080 bytes 2024-09-12T16:01:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1554409472 bytes 2024-09-12T16:01:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1555006464 bytes 2024-09-12T16:01:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1562668032 bytes 2024-09-12T16:01:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1597154304 bytes 2024-09-12T16:01:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1619239936 bytes 2024-09-12T16:01:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1662326784 bytes 2024-09-12T16:01:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1720974336 bytes 2024-09-12T16:01:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1747507200 bytes 2024-09-12T16:01:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1791241216 bytes 2024-09-12T16:01:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1814169600 bytes 2024-09-12T16:01:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1851516928 bytes 2024-09-12T16:01:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1873632256 bytes 2024-09-12T16:01:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1931491328 bytes 2024-09-12T16:01:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1944872960 bytes 2024-09-12T16:01:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1962008576 bytes 2024-09-12T16:01:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1964398592 bytes 2024-09-12T16:01:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2075525120 bytes 2024-09-12T16:01:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2101874688 bytes 2024-09-12T16:01:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2144615424 bytes 2024-09-12T16:01:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2199926784 bytes 2024-09-12T16:01:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2216955904 bytes 2024-09-12T16:01:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2293624832 bytes 2024-09-12T16:01:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2337234944 bytes 2024-09-12T16:01:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2368884736 bytes 2024-09-12T16:01:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2416529408 bytes 2024-09-12T16:01:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2467138560 bytes 2024-09-12T16:01:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2481952768 bytes 2024-09-12T16:01:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2525216768 bytes 2024-09-12T16:01:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2567164928 bytes 2024-09-12T16:01:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2605307904 bytes 2024-09-12T16:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2665937920 bytes 2024-09-12T16:01:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2699840512 bytes 2024-09-12T16:01:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2780190720 bytes 2024-09-12T16:01:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2827481088 bytes 2024-09-12T16:01:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2876836864 bytes 2024-09-12T16:01:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2915050496 bytes 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:01:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:01:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:01:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:01:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T16:01:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":17843507.2,"bytes_per_snapshot":536081408.0,"last_snapshot_bytes":535305216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156853,"remote_snapshot_timestamp":1726156853,"replay_state":"idle"} 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:01:06 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:00:51' 2024-09-12T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T16:01:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T16:01:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T16:01:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":17843507.2,"bytes_per_snapshot":536081408.0,"last_snapshot_bytes":535305216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156853,"remote_snapshot_timestamp":1726156853,"replay_state":"idle"} 2024-09-12T16:01:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:01:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:01:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T16:01:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:01:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T16:01:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T16:01:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T16:01:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T16:01:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:01:06' 2024-09-12T16:01:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 16:01:06' 2024-09-12T16:01:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:01:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 16:01:06' 2024-09-12T16:01:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:01:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:01:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T16:01:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOK44284431mirror.0smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":17843507.2,"bytes_per_snapshot":536081408.0,"last_snapshot_bytes":535305216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156853,"remote_snapshot_timestamp":1726156853,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 16:01:06cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 16:00:51' 2024-09-12T16:01:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T16:01:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T16:01:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:01:06' 2024-09-12T16:01:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T16:01:14.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T16:01:14.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:01:14.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T16:01:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap6 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap6 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap6 2024-09-12T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:01:23.350 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5075778773138281) > weight (0.25) 2024-09-12T16:01:23.351 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3740508021131992) > weight (0.25) 2024-09-12T16:01:23.351 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-12T16:01:23.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-12T16:01:23.351 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-12T16:01:23.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:01:23.352+0000 7f386fc17640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 41879) UID: 1000 2024-09-12T16:01:25.352 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-12T16:01:25.353 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:01:25.353 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-12T16:01:25.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-12T16:01:25.353 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-12T16:01:25.357 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-12T16:01:25.357 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-09-12T16:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T16:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:01:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:01:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T16:01:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:01:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:01:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:01:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:01:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:01:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:01:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:01:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:01:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:01:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:01:45.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:01:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:01:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:01:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T16:01:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T16:01:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:01:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:01:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:01:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T16:01:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:01:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:01:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:01:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:01:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:01:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:01:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:01:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:01:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:01:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:01:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T16:01:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:01:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:01:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:01:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:01:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:01:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:01:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:01:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:01:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:01:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:01:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:01:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T16:01:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T16:01:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:01:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:01:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T16:01:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T16:01:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:01:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T16:01:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:01:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 20 2024-09-12T16:01:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:01:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:01:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:01:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:01:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:01:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=20 2024-09-12T16:01:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:01:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:01:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:01:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:01:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:01:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:01:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:01:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:01:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:01:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:01:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:01:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 20 = '' 2024-09-12T16:01:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:01:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:01:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:01:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:01:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:01:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:01:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:01:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:01:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:01:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:01:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 20 = '' 2024-09-12T16:01:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:01:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:01:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:01:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:01:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:01:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:01:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:01:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:01:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:01:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:01:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 20 = '' 2024-09-12T16:01:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:01:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:01:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:01:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:01:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:01:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:01:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=20 2024-09-12T16:01:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 20 = 20 2024-09-12T16:01:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-12T16:01:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:01:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap6 2024-09-12T16:01:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:01:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:01:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:01:51.348 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T16:01:51.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T16:01:51.349 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-12T16:01:51.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:01:51.349+0000 7f88e1336640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41361) UID: 1000 2024-09-12T16:01:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-12T16:01:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:01:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:01:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:01:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 12 2024-09-12T16:01:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:01:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:01:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:01:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:01:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:01:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:01:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:01:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:00:05 2024 2024-09-12T16:01:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:01:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:01:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:01:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:01:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:01:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap5-clean ']' 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap5-clean 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:01:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:01:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:52.350 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T16:01:52.350 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:01:52.351 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T16:01:52.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T16:01:52.351 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T16:01:52.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T16:01:52.354 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-12T16:01:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:01:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap6 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap6 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:01:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:01:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:01:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap6-clean 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap6-clean 2024-09-12T16:01:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T16:01:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:01:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T16:01:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:01:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:01:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:01:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:01:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:01:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:01:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T16:01:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 39300096 bytes 2024-09-12T16:01:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 80394240 bytes 2024-09-12T16:01:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 80994304 bytes 2024-09-12T16:01:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 109122560 bytes 2024-09-12T16:01:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 127068160 bytes 2024-09-12T16:01:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 189369344 bytes 2024-09-12T16:01:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 216492032 bytes 2024-09-12T16:01:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 257901568 bytes 2024-09-12T16:01:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 293936128 bytes 2024-09-12T16:01:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 317557760 bytes 2024-09-12T16:01:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 359404544 bytes 2024-09-12T16:01:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 362943488 bytes 2024-09-12T16:01:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 363570176 bytes 2024-09-12T16:01:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 384129024 bytes 2024-09-12T16:01:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 431797248 bytes 2024-09-12T16:01:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 470918144 bytes 2024-09-12T16:01:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 474181632 bytes 2024-09-12T16:01:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 523794432 bytes 2024-09-12T16:01:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 553943040 bytes 2024-09-12T16:01:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 603536384 bytes 2024-09-12T16:01:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 648814592 bytes 2024-09-12T16:01:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 678213632 bytes 2024-09-12T16:01:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 700662784 bytes 2024-09-12T16:01:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 746397696 bytes 2024-09-12T16:01:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 790137856 bytes 2024-09-12T16:01:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 850514944 bytes 2024-09-12T16:01:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 884508672 bytes 2024-09-12T16:01:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 965129216 bytes 2024-09-12T16:01:59.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 985325568 bytes 2024-09-12T16:01:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1019760640 bytes 2024-09-12T16:01:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1039697920 bytes 2024-09-12T16:01:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1076677632 bytes 2024-09-12T16:02:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1122071552 bytes 2024-09-12T16:02:00.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1162222592 bytes 2024-09-12T16:02:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1212478464 bytes 2024-09-12T16:02:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1275404288 bytes 2024-09-12T16:02:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1336586240 bytes 2024-09-12T16:02:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1419354112 bytes 2024-09-12T16:02:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1501216768 bytes 2024-09-12T16:02:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1548722176 bytes 2024-09-12T16:02:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1602284544 bytes 2024-09-12T16:02:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1636389888 bytes 2024-09-12T16:02:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1679354880 bytes 2024-09-12T16:02:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1680001024 bytes 2024-09-12T16:02:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1703004160 bytes 2024-09-12T16:02:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1731664896 bytes 2024-09-12T16:02:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1737567232 bytes 2024-09-12T16:02:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1763607552 bytes 2024-09-12T16:02:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1806756864 bytes 2024-09-12T16:02:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1866537984 bytes 2024-09-12T16:02:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 1887596544 bytes 2024-09-12T16:02:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 1888216064 bytes 2024-09-12T16:02:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 1919589376 bytes 2024-09-12T16:02:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 1984450560 bytes 2024-09-12T16:02:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2044210176 bytes 2024-09-12T16:02:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2050644992 bytes 2024-09-12T16:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2051254272 bytes 2024-09-12T16:02:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2109845504 bytes 2024-09-12T16:02:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2156877824 bytes 2024-09-12T16:02:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:02:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:02:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:02:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:02:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T16:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:02:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:02:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:02:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:02:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T16:02:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536081408.0,"last_snapshot_bytes":535305216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156853,"remote_snapshot_timestamp":1726156853,"replay_state":"idle"} 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:01:36 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:01:51' 2024-09-12T16:02:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T16:02:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T16:02:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T16:02:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536081408.0,"last_snapshot_bytes":535305216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156853,"remote_snapshot_timestamp":1726156853,"replay_state":"idle"} 2024-09-12T16:02:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:02:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:02:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T16:02:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:02:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T16:02:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T16:02:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T16:02:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T16:02:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:01:36' 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 16:01:36' 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 16:01:36' 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:02:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T16:02:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOK45204523mirror.0smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536081408.0,"last_snapshot_bytes":535305216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156853,"remote_snapshot_timestamp":1726156853,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 16:01:36cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 16:01:51' 2024-09-12T16:02:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T16:02:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T16:02:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:01:36' 2024-09-12T16:02:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T16:02:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T16:02:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:02:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T16:02:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:02:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap7 2024-09-12T16:02:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap7 2024-09-12T16:02:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap7 2024-09-12T16:02:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:02:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:02:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:02:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T16:02:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:02:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:02:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:02:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:02:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:02:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:02:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:02:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:02:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:02:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:02:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:02:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:02:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:02:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:02:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:02:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:02:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:02:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:02:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:02:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:02:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:02:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T16:02:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T16:02:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:02:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:02:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:02:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T16:02:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:02:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:02:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:02:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:02:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:02:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:02:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:02:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T16:02:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:02:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:02:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:02:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:02:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:02:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:02:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:02:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:02:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T16:02:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T16:02:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 23 2024-09-12T16:02:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:02:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:02:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:02:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:02:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:02:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=23 2024-09-12T16:02:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:02:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:02:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:02:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:02:38.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:02:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:02:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:02:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:02:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:02:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:02:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:02:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 23 = '' 2024-09-12T16:02:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:02:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:02:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:02:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:02:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:02:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:02:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:02:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:02:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:02:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:02:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 23 = '' 2024-09-12T16:02:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:02:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:02:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:02:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:02:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:02:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 23 = '' 2024-09-12T16:02:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:02:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:02:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:02:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:02:41.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:02:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:02:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=23 2024-09-12T16:02:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 23 = 23 2024-09-12T16:02:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-12T16:02:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap7 2024-09-12T16:02:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:02:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:02:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:02:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-12T16:02:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:02:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 14 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:01:52 2024 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:02:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:02:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:02:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap6-clean ']' 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap6-clean 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:02:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:02:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:02:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap7 2024-09-12T16:02:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap7 2024-09-12T16:02:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:02:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:02:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:02:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:02:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:02:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:02:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap7-clean 2024-09-12T16:02:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-12T16:02:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap7-clean 2024-09-12T16:02:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T16:02:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:02:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T16:02:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:02:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=6 2024-09-12T16:02:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:02:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T16:02:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 21640192 bytes 2024-09-12T16:02:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 55430144 bytes 2024-09-12T16:02:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 112538624 bytes 2024-09-12T16:02:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 153272320 bytes 2024-09-12T16:02:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 185023488 bytes 2024-09-12T16:02:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 186940416 bytes 2024-09-12T16:02:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 204454912 bytes 2024-09-12T16:02:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 279336960 bytes 2024-09-12T16:02:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 335315968 bytes 2024-09-12T16:02:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 346714112 bytes 2024-09-12T16:02:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 429570048 bytes 2024-09-12T16:02:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 440074240 bytes 2024-09-12T16:02:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 457473024 bytes 2024-09-12T16:02:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 479556608 bytes 2024-09-12T16:02:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 506511360 bytes 2024-09-12T16:02:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 534881280 bytes 2024-09-12T16:02:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 590695424 bytes 2024-09-12T16:02:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 675731456 bytes 2024-09-12T16:02:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 710274048 bytes 2024-09-12T16:02:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 747051008 bytes 2024-09-12T16:02:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 778905600 bytes 2024-09-12T16:02:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 830242816 bytes 2024-09-12T16:02:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 846809088 bytes 2024-09-12T16:02:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 900149248 bytes 2024-09-12T16:02:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 938439680 bytes 2024-09-12T16:02:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1017228288 bytes 2024-09-12T16:02:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1113899008 bytes 2024-09-12T16:02:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1182930944 bytes 2024-09-12T16:02:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1189632000 bytes 2024-09-12T16:02:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1190238208 bytes 2024-09-12T16:02:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1278671872 bytes 2024-09-12T16:02:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1279310848 bytes 2024-09-12T16:02:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1338986496 bytes 2024-09-12T16:02:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1362890752 bytes 2024-09-12T16:02:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1380700160 bytes 2024-09-12T16:02:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1407939584 bytes 2024-09-12T16:02:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1436816384 bytes 2024-09-12T16:02:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1448692736 bytes 2024-09-12T16:02:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:02:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:02:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:02:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T16:02:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:02:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:02:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:02:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:02:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:02:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T16:02:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:02:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:02:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:02:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:02:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:02:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532194304.0,"last_snapshot_bytes":529083392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156906,"remote_snapshot_timestamp":1726156906,"replay_state":"idle"} 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:02:36 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T16:02:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:02:51' 2024-09-12T16:02:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T16:02:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T16:02:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T16:02:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532194304.0,"last_snapshot_bytes":529083392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156906,"remote_snapshot_timestamp":1726156906,"replay_state":"idle"} 2024-09-12T16:02:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:02:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:02:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T16:02:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:02:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T16:02:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T16:02:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T16:02:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T16:02:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:02:36' 2024-09-12T16:02:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 16:02:36' 2024-09-12T16:02:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:02:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:02:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 16:02:36' 2024-09-12T16:02:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:02:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:02:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T16:02:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOK45204523mirror.0smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532194304.0,"last_snapshot_bytes":529083392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726156906,"remote_snapshot_timestamp":1726156906,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 16:02:36cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 16:02:51' 2024-09-12T16:02:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T16:02:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T16:02:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:02:36' 2024-09-12T16:02:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T16:02:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T16:02:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:02:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T16:02:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap8 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap8 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:02:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap8 2024-09-12T16:02:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:03:19.358 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8635096716582148) > weight (0.25) 2024-09-12T16:03:19.358 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9430313877840548) > weight (0.25) 2024-09-12T16:03:19.358 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-12T16:03:19.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-12T16:03:19.359 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-12T16:03:19.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:03:19.359+0000 7f68d4ba5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42505) UID: 1000 2024-09-12T16:03:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T16:03:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:03:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:03:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T16:03:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:03:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:03:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:03:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:03:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:03:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:03:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:03:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:03:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:03:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:03:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:03:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:03:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:03:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:03:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:03:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:03:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:03:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:03:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:03:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:03:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:03:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:03:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T16:03:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T16:03:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:03:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:03:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:03:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:03:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:03:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:03:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:03:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:03:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T16:03:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:03:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:03:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:03:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:03:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:03:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:03:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:03:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:03:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T16:03:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T16:03:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:03:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:03:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T16:03:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T16:03:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:03:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T16:03:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:03:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:03:26.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 26 2024-09-12T16:03:26.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4337571239420539) > weight (0.25) 2024-09-12T16:03:26.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8476467388092039) > weight (0.25) 2024-09-12T16:03:26.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8316183569383856) > weight (0.25) 2024-09-12T16:03:26.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.25310482777512044) > weight (0.25) 2024-09-12T16:03:26.355 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-12T16:03:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:03:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:03:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:03:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:03:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:03:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:03:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=26 2024-09-12T16:03:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:03:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:03:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:03:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:03:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:03:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:03:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:03:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:03:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:03:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:03:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:03:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 26 = '' 2024-09-12T16:03:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:03:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:03:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:03:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:03:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:03:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 26 = '' 2024-09-12T16:03:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:03:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:03:27.360 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-12T16:03:27.360 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:03:27.360 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-12T16:03:27.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-12T16:03:27.361 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-12T16:03:27.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-12T16:03:27.365 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-09-12T16:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:03:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:03:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:03:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:03:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:03:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:03:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:03:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:03:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 26 = '' 2024-09-12T16:03:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:03:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:03:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:03:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:03:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:03:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:03:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=26 2024-09-12T16:03:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 26 = 26 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap8 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:03:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:03:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-12T16:03:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:03:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:03:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 16 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:01:52 2024 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:03:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:03:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:03:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:03:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap7-clean ']' 2024-09-12T16:03:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-12T16:03:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap7-clean 2024-09-12T16:03:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:03:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:03:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:03:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:03:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:03:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap8 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap8 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:03:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:03:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:03:34.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:03:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap8-clean 2024-09-12T16:03:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-12T16:03:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:03:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap8-clean 2024-09-12T16:03:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T16:03:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:03:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T16:03:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:03:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:03:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=8 2024-09-12T16:03:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:03:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T16:03:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 68344832 bytes 2024-09-12T16:03:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 89132032 bytes 2024-09-12T16:03:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 171021312 bytes 2024-09-12T16:03:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 192750592 bytes 2024-09-12T16:03:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 213743616 bytes 2024-09-12T16:03:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 289426432 bytes 2024-09-12T16:03:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 298150912 bytes 2024-09-12T16:03:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 328406016 bytes 2024-09-12T16:03:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 365477888 bytes 2024-09-12T16:03:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 445667328 bytes 2024-09-12T16:03:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 468967424 bytes 2024-09-12T16:03:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 524662784 bytes 2024-09-12T16:03:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 579509248 bytes 2024-09-12T16:03:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 662028288 bytes 2024-09-12T16:03:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 715352064 bytes 2024-09-12T16:03:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 777798656 bytes 2024-09-12T16:03:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 821707776 bytes 2024-09-12T16:03:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 875153408 bytes 2024-09-12T16:03:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 913746944 bytes 2024-09-12T16:03:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 952363008 bytes 2024-09-12T16:03:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1018293248 bytes 2024-09-12T16:03:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1028219904 bytes 2024-09-12T16:03:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1070244864 bytes 2024-09-12T16:03:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1085717504 bytes 2024-09-12T16:03:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1101880320 bytes 2024-09-12T16:03:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1127430144 bytes 2024-09-12T16:03:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1185504256 bytes 2024-09-12T16:03:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1189288960 bytes 2024-09-12T16:03:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1200778240 bytes 2024-09-12T16:03:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1208871936 bytes 2024-09-12T16:03:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1260417024 bytes 2024-09-12T16:03:40.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1276321792 bytes 2024-09-12T16:03:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1320557568 bytes 2024-09-12T16:03:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1321178112 bytes 2024-09-12T16:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1375048704 bytes 2024-09-12T16:03:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1421660160 bytes 2024-09-12T16:03:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1501036544 bytes 2024-09-12T16:03:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1587336192 bytes 2024-09-12T16:03:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1627045888 bytes 2024-09-12T16:03:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1698034688 bytes 2024-09-12T16:03:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1710823424 bytes 2024-09-12T16:03:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1778685952 bytes 2024-09-12T16:03:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1826908160 bytes 2024-09-12T16:03:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1864508416 bytes 2024-09-12T16:03:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1943882752 bytes 2024-09-12T16:03:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2024774656 bytes 2024-09-12T16:03:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2112548864 bytes 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:03:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:03:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:03:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:03:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:03:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T16:03:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:03:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:03:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:03:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:03:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:03:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":16568832.0,"bytes_per_snapshot":513784320.0,"last_snapshot_bytes":497064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726157005,"remote_snapshot_timestamp":1726157005,"replay_state":"idle"} 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:03:36 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T16:03:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T16:03:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:03:21' 2024-09-12T16:03:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T16:03:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T16:03:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T16:03:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":16568832.0,"bytes_per_snapshot":513784320.0,"last_snapshot_bytes":497064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726157005,"remote_snapshot_timestamp":1726157005,"replay_state":"idle"} 2024-09-12T16:03:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:03:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:03:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T16:03:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:03:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T16:03:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T16:03:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T16:03:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T16:03:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:03:36' 2024-09-12T16:03:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 16:03:36' 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 16:03:36' 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T16:03:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOK45204523mirror.0smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":16568832.0,"bytes_per_snapshot":513784320.0,"last_snapshot_bytes":497064960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726157005,"remote_snapshot_timestamp":1726157005,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 16:03:36cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 16:03:21' 2024-09-12T16:03:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T16:03:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T16:03:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:03:36' 2024-09-12T16:03:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T16:03:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T16:03:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:03:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T16:03:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap9 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap9 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap9 2024-09-12T16:03:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:04:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T16:04:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:04:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:04:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:04:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:04:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:04:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:04:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:04:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:04:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:04:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:04:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:04:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:04:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:04:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:04:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:04:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:04:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T16:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T16:04:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:04:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:04:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:04:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:04:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:04:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:04:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:04:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:04:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T16:04:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:04:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:04:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:04:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:04:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:04:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T16:04:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:04:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:04:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T16:04:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T16:04:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:04:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T16:04:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:04:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:04:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 29 2024-09-12T16:04:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:04:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:04:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:04:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:04:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:04:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=29 2024-09-12T16:04:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:04:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:04:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:04:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:04:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:04:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:04:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 29 = '' 2024-09-12T16:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:04:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:04:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:04:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:04:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:04:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:04:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:04:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:04:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:04:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:04:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 29 = '' 2024-09-12T16:04:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:04:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:04:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:04:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:04:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:04:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:04:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:04:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:04:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:04:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:04:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 29 = '' 2024-09-12T16:04:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:04:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:04:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:04:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:04:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:04:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=29 2024-09-12T16:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 29 = 29 2024-09-12T16:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-12T16:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap9 2024-09-12T16:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:04:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:04:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 18 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:04:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:03:31 2024 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:04:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:04:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap8-clean ']' 2024-09-12T16:04:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap8-clean 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:04:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:04:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:04:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap9 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap9 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:04:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:04:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:04:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:04:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap9-clean 2024-09-12T16:04:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-12T16:04:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:04:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap9-clean 2024-09-12T16:04:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T16:04:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:04:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror test 2024-09-12T16:04:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:04:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:04:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=13 2024-09-12T16:04:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:04:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-12T16:04:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 11493376 bytes 2024-09-12T16:04:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 33980416 bytes 2024-09-12T16:04:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 93921280 bytes 2024-09-12T16:04:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 153296896 bytes 2024-09-12T16:04:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 153904128 bytes 2024-09-12T16:04:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 169322496 bytes 2024-09-12T16:04:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 294580224 bytes 2024-09-12T16:04:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 318292992 bytes 2024-09-12T16:04:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 355670016 bytes 2024-09-12T16:04:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 456076288 bytes 2024-09-12T16:04:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 474528768 bytes 2024-09-12T16:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 522721280 bytes 2024-09-12T16:04:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 607707136 bytes 2024-09-12T16:04:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 633183232 bytes 2024-09-12T16:04:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 644980736 bytes 2024-09-12T16:04:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 673720320 bytes 2024-09-12T16:04:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 707273728 bytes 2024-09-12T16:04:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 762562560 bytes 2024-09-12T16:04:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 763174912 bytes 2024-09-12T16:04:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 768864256 bytes 2024-09-12T16:04:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 771550208 bytes 2024-09-12T16:04:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 851369984 bytes 2024-09-12T16:04:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 870467584 bytes 2024-09-12T16:04:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 938092544 bytes 2024-09-12T16:04:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 944540672 bytes 2024-09-12T16:04:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 946130944 bytes 2024-09-12T16:04:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 978985984 bytes 2024-09-12T16:04:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1004448768 bytes 2024-09-12T16:04:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1045570560 bytes 2024-09-12T16:04:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1102928896 bytes 2024-09-12T16:04:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1152333824 bytes 2024-09-12T16:04:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1167321088 bytes 2024-09-12T16:04:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1223894016 bytes 2024-09-12T16:04:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1275171840 bytes 2024-09-12T16:04:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1298579456 bytes 2024-09-12T16:04:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1387966464 bytes 2024-09-12T16:04:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1432646656 bytes 2024-09-12T16:04:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1456371712 bytes 2024-09-12T16:04:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1518660608 bytes 2024-09-12T16:04:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1551694848 bytes 2024-09-12T16:04:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1577362432 bytes 2024-09-12T16:04:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1608854528 bytes 2024-09-12T16:04:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1629571072 bytes 2024-09-12T16:04:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1657003008 bytes 2024-09-12T16:04:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1686607872 bytes 2024-09-12T16:04:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1750940672 bytes 2024-09-12T16:04:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1806155776 bytes 2024-09-12T16:04:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1808712704 bytes 2024-09-12T16:04:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1856128000 bytes 2024-09-12T16:04:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1896872960 bytes 2024-09-12T16:04:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 1951509504 bytes 2024-09-12T16:04:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 1980559360 bytes 2024-09-12T16:04:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2016795648 bytes 2024-09-12T16:04:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2123380736 bytes 2024-09-12T16:04:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2173602816 bytes 2024-09-12T16:04:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2231041024 bytes 2024-09-12T16:04:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2322665472 bytes 2024-09-12T16:04:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2359216128 bytes 2024-09-12T16:04:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2416009216 bytes 2024-09-12T16:04:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2421751808 bytes 2024-09-12T16:04:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2448455680 bytes 2024-09-12T16:04:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2545274880 bytes 2024-09-12T16:04:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2566492160 bytes 2024-09-12T16:04:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2567097344 bytes 2024-09-12T16:04:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2605253632 bytes 2024-09-12T16:04:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2605874176 bytes 2024-09-12T16:04:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2631577600 bytes 2024-09-12T16:04:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2681807872 bytes 2024-09-12T16:04:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2685784064 bytes 2024-09-12T16:04:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2718576640 bytes 2024-09-12T16:04:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 2744709120 bytes 2024-09-12T16:04:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 2807306240 bytes 2024-09-12T16:04:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 2889757696 bytes 2024-09-12T16:04:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 2916231168 bytes 2024-09-12T16:04:37.356 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T16:04:37.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T16:04:37.357 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-12T16:04:37.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:04:37.357+0000 7fb3e2497640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42834) UID: 1000 2024-09-12T16:04:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 2956769280 bytes 2024-09-12T16:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7600 IOs, 3013106688 bytes 2024-09-12T16:04:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7700 IOs, 3036436480 bytes 2024-09-12T16:04:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7800 IOs, 3062429696 bytes 2024-09-12T16:04:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7900 IOs, 3063038976 bytes 2024-09-12T16:04:38.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8000 IOs, 3099675648 bytes 2024-09-12T16:04:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:04:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:04:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:04:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:04:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:04:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-12T16:04:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:04:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local description_pattern= 2024-09-12T16:04:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local service_pattern= 2024-09-12T16:04:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:04:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:04:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-12T16:04:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='test: 2024-09-12T16:04:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: global_id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:04:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+replaying 2024-09-12T16:04:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: replaying, {"bytes_per_second":17452305.07,"bytes_per_snapshot":510317056.0,"last_snapshot_bytes":523569152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726157056,"remote_snapshot_timestamp":1726157056,"replay_state":"idle"} 2024-09-12T16:04:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: service: mirror.1 on smithi019 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:04:36 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: peer_sites: 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: name: cluster2 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: state: up+stopped 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: description: local image is primary 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: last_update: 2024-09-12 16:04:21' 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ state: .*up+replaying' 2024-09-12T16:04:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: state: up+replaying 2024-09-12T16:04:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^ description: .*' 2024-09-12T16:04:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: description: replaying, {"bytes_per_second":17452305.07,"bytes_per_snapshot":510317056.0,"last_snapshot_bytes":523569152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726157056,"remote_snapshot_timestamp":1726157056,"replay_state":"idle"} 2024-09-12T16:04:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:04:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:04:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep '^up+' 2024-09-12T16:04:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:04:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep 'service: *mirror..* on ' 2024-09-12T16:04:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: service: mirror.1 on smithi019 2024-09-12T16:04:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update 2024-09-12T16:04:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:04:36' 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-12 16:04:36' 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state_pattern=up+replaying 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local 'prev_last_update=2024-09-12 16:04:36' 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status 2024-09-12T16:04:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ CEPH_ARGS= 2024-09-12T16:04:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-12T16:04:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status='OKOKOK142154261mirror.1smithi01919.3.0-4883-gd61b0886trueOK42184264mirror.2smithi01919.3.0-4883-gd61b0886falseOK43584361mirror.3smithi01919.3.0-4883-gd61b0886falseOKtesta60bbc86-ef92-42aa-baad-7368fc65517bup+replayingreplaying, {"bytes_per_second":17452305.07,"bytes_per_snapshot":510317056.0,"last_snapshot_bytes":523569152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726157056,"remote_snapshot_timestamp":1726157056,"replay_state":"idle"}42154261mirror.1smithi0192024-09-12 16:04:36cluster25d012f07-fdca-4a2f-a7fc-86d6406006b8up+stoppedlocal image is primary2024-09-12 16:04:21' 2024-09-12T16:04:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_update state 2024-09-12T16:04:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-12T16:04:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ last_update='2024-09-12 16:04:36' 2024-09-12T16:04:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-12T16:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=up+replaying 2024-09-12T16:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo up+replaying 2024-09-12T16:04:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep up+replaying 2024-09-12T16:04:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:up+replaying 2024-09-12T16:04:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:04:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap10 2024-09-12T16:04:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap10 2024-09-12T16:04:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:04:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:04:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:04:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap10 2024-09-12T16:04:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:04:44.358 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T16:04:44.358 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:04:44.358 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T16:04:44.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T16:04:44.359 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T16:04:44.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T16:04:44.362 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-09-12T16:05:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:05:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:05:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T16:05:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:05:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:05:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:05:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T16:05:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:05:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:05:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:05:10.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:05:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:05:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:05:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:05:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:05:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T16:05:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T16:05:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:05:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:05:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:05:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:05:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:05:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:05:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:05:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:05:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T16:05:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:05:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:05:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:05:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:05:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:05:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T16:05:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T16:05:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:05:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 32 2024-09-12T16:05:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:05:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:05:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:05:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:05:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=32 2024-09-12T16:05:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:05:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:05:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:05:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:05:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:05:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:05:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:05:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:05:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:05:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:05:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 32 = '' 2024-09-12T16:05:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:05:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:05:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:05:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:05:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:05:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:05:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:05:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:05:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:05:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 32 = '' 2024-09-12T16:05:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:05:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:05:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:05:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:05:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:05:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:05:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:05:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:05:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:05:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 32 = '' 2024-09-12T16:05:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:05:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:05:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:05:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=32 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 32 = 32 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap10 2024-09-12T16:05:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:05:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:05:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:05:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-12T16:05:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 20 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:05:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:05:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:03:31 2024 2024-09-12T16:05:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:05:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:05:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:05:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:05:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:05:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:05:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snap9-clean ']' 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap9-clean 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap10 2024-09-12T16:05:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap10 2024-09-12T16:05:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:19.366 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6690789649432202) > weight (0.25) 2024-09-12T16:05:19.366 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4698678645564762) > weight (0.25) 2024-09-12T16:05:19.366 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-12T16:05:19.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-12T16:05:19.367 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-12T16:05:19.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:05:19.366+0000 7fdf652bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 43642) UID: 1000 2024-09-12T16:05:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ clean_snap_name=snap10-clean 2024-09-12T16:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-12T16:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap10-clean 2024-09-12T16:05:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-12T16:05:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:05:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:05:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-12T16:05:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:05:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:05:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:05:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:05:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:05:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:05:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:05:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:05:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:05:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:05:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:05:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:05:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:05:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-12T16:05:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-12T16:05:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:05:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:05:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:05:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:05:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:05:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:05:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:05:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-12T16:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:05:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:05:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:05:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:05:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:05:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:05:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:05:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:05:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:05:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-test.status 2024-09-12T16:05:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-test.status 2024-09-12T16:05:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:05:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:05:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-test.status 2024-09-12T16:05:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-test.status 2024-09-12T16:05:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:05:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-12T16:05:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-12T16:05:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 34 2024-09-12T16:05:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:05:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:05:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.status 2024-09-12T16:05:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-12T16:05:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:05:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=34 2024-09-12T16:05:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:05:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:05:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:05:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:05:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:05:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:05:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:05:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:05:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:05:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:05:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 34 = '' 2024-09-12T16:05:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:05:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:05:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.status 2024-09-12T16:05:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-12T16:05:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:05:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:05:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=34 2024-09-12T16:05:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 34 = 34 2024-09-12T16:05:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-12T16:05:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:05:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap10-clean 2024-09-12T16:05:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:05:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:05:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:05:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-12T16:05:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'test': 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 21 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f00cd518 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f00cd518 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:05:17 2024 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 15:56:38 2024 2024-09-12T16:05:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:05:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:05:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:05:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a60bbc86-ef92-42aa-baad-7368fc65517b 2024-09-12T16:05:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:05:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 10 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap1 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap1 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap1 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap1-clean 2024-09-12T16:05:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-12T16:05:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap1-clean 2024-09-12T16:05:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:27.368 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-12T16:05:27.368 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:05:27.368 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-12T16:05:27.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-12T16:05:27.368 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-12T16:05:27.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-12T16:05:27.371 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-12T16:05:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap2 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap2 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap2 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap2-clean 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap2-clean 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap3 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap3 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap3 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap3-clean 2024-09-12T16:05:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-12T16:05:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap3-clean 2024-09-12T16:05:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap4 2024-09-12T16:05:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap4 2024-09-12T16:05:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap4 2024-09-12T16:05:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap4-clean 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap4-clean 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap5 2024-09-12T16:05:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap5 2024-09-12T16:05:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap5 2024-09-12T16:05:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap5-clean 2024-09-12T16:05:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-12T16:05:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap5-clean 2024-09-12T16:05:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap6 2024-09-12T16:05:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap6 2024-09-12T16:05:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap6 2024-09-12T16:05:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap6-clean 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap6-clean 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap7 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap7 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap7 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap7-clean 2024-09-12T16:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-12T16:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap7-clean 2024-09-12T16:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap8 2024-09-12T16:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap8 2024-09-12T16:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap8 2024-09-12T16:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap8-clean 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap8-clean 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap9 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap9 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap9 2024-09-12T16:05:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap9-clean 2024-09-12T16:05:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-12T16:05:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap9-clean 2024-09-12T16:05:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap10 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap10 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap10 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap10-clean 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap10-clean 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:05:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export 2024-09-12T16:05:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:05:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:06:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-12T16:06:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:06:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-test.export /tmp/tmp.65UL20DnMX/cluster1-mirror-test.export 2024-09-12T16:06:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 10 2024-09-12T16:06:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap1 2024-09-12T16:06:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-12T16:06:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap1 2024-09-12T16:06:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-12T16:06:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap1-clean 2024-09-12T16:06:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-12T16:06:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap1-clean 2024-09-12T16:06:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-12T16:06:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap2 2024-09-12T16:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-12T16:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap2 2024-09-12T16:06:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-12T16:06:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap2-clean 2024-09-12T16:06:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-12T16:06:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap2-clean 2024-09-12T16:06:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-12T16:06:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap3 2024-09-12T16:06:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-12T16:06:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap3 2024-09-12T16:06:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-12T16:06:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap3-clean 2024-09-12T16:06:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-12T16:06:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap3-clean 2024-09-12T16:06:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-12T16:06:06.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap4 2024-09-12T16:06:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-12T16:06:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap4 2024-09-12T16:06:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-12T16:06:07.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap4-clean 2024-09-12T16:06:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-12T16:06:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap4-clean 2024-09-12T16:06:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-12T16:06:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap5 2024-09-12T16:06:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-12T16:06:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap5 2024-09-12T16:06:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-12T16:06:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap5-clean 2024-09-12T16:06:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-12T16:06:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap5-clean 2024-09-12T16:06:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-12T16:06:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap6 2024-09-12T16:06:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-12T16:06:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap6 2024-09-12T16:06:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-12T16:06:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap6-clean 2024-09-12T16:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-12T16:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap6-clean 2024-09-12T16:06:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-12T16:06:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap7 2024-09-12T16:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-12T16:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap7 2024-09-12T16:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-12T16:06:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap7-clean 2024-09-12T16:06:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-12T16:06:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap7-clean 2024-09-12T16:06:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-12T16:06:13.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap8 2024-09-12T16:06:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-12T16:06:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap8 2024-09-12T16:06:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-12T16:06:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap8-clean 2024-09-12T16:06:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-12T16:06:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap8-clean 2024-09-12T16:06:14.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-12T16:06:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap9 2024-09-12T16:06:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-12T16:06:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap9 2024-09-12T16:06:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-12T16:06:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap9-clean 2024-09-12T16:06:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-12T16:06:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap9-clean 2024-09-12T16:06:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-12T16:06:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap10 2024-09-12T16:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-12T16:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap10 2024-09-12T16:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-12T16:06:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap10-clean 2024-09-12T16:06:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-12T16:06:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap10-clean 2024-09-12T16:06:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-12T16:06:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:06:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror test 2024-09-12T16:06:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:06:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:06:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror test 2024-09-12T16:06:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-12T16:06:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-12T16:06:19.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:06:19.542+0000 7f51469e8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e23db189c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:06:19.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:06:19.542+0000 7f514d1f5640 -1 rbd::mirror::ImageReplayer: 0x55e23d80ca00 [3/a60bbc86-ef92-42aa-baad-7368fc65517b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:06:21.494 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:06:21.493+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b121dc70 2024-09-12T16:06:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:06:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror test 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=test 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-12T16:06:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:06:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-12T16:06:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:06:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:06:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:06:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-12T16:06:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:06:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=test 2024-09-12T16:06:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:06:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror test 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=test 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-12T16:06:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:06:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-12T16:06:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:06:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:06:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:06:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ testlog 'TEST: create many images' 2024-09-12T16:06:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ tee -a /tmp/tmp.65UL20DnMX/rbd-mirror.test.log 2024-09-12T16:06:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ date '+%F %T' 2024-09-12T16:06:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo 2024-09-12 16:06:21 TEST: create many images 2024-09-12T16:06:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:2024-09-12 16:06:21 TEST: create many images 2024-09-12T16:06:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ snap_name=snap 2024-09-12T16:06:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 50 2024-09-12T16:06:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:06:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_1 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-12T16:06:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:06:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-12T16:06:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_1': 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1264a4006669 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1264a4006669 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:06:21 2024 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:06:21 2024 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:06:21 2024 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:06:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:06:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 135c0a36-496a-4b49-bf12-e55a9a725020 2024-09-12T16:06:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:06:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:06:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-12T16:06:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:06:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:06:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:06:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:06:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:06:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 32305152 bytes 2024-09-12T16:06:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 46672896 bytes 2024-09-12T16:06:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 69183488 bytes 2024-09-12T16:06:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 118533120 bytes 2024-09-12T16:06:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 165626880 bytes 2024-09-12T16:06:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 205739008 bytes 2024-09-12T16:06:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 219156480 bytes 2024-09-12T16:06:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 279176192 bytes 2024-09-12T16:06:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 350841856 bytes 2024-09-12T16:06:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 419026944 bytes 2024-09-12T16:06:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 491644928 bytes 2024-09-12T16:06:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 528216064 bytes 2024-09-12T16:06:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 528820224 bytes 2024-09-12T16:06:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 577970176 bytes 2024-09-12T16:06:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 616349696 bytes 2024-09-12T16:06:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 640777216 bytes 2024-09-12T16:06:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 690531328 bytes 2024-09-12T16:06:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 718115840 bytes 2024-09-12T16:06:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 734517248 bytes 2024-09-12T16:06:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 823450624 bytes 2024-09-12T16:06:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 855185408 bytes 2024-09-12T16:06:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 915780608 bytes 2024-09-12T16:06:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 948580352 bytes 2024-09-12T16:06:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 949189632 bytes 2024-09-12T16:06:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1019948032 bytes 2024-09-12T16:06:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1074893824 bytes 2024-09-12T16:06:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1118734336 bytes 2024-09-12T16:06:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1152799744 bytes 2024-09-12T16:06:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1197529088 bytes 2024-09-12T16:06:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1227132928 bytes 2024-09-12T16:06:27.372 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-12T16:06:27.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-12T16:06:27.373 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-12T16:06:27.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:06:27.374+0000 7f67b0c20640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40871) UID: 1000 2024-09-12T16:06:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1323065344 bytes 2024-09-12T16:06:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1356452864 bytes 2024-09-12T16:06:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1357067264 bytes 2024-09-12T16:06:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1376113664 bytes 2024-09-12T16:06:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1458808832 bytes 2024-09-12T16:06:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1498914816 bytes 2024-09-12T16:06:28.373 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-12T16:06:28.373 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:06:28.374 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-12T16:06:28.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-12T16:06:28.374 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-12T16:06:28.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-12T16:06:28.378 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-12T16:06:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1586221056 bytes 2024-09-12T16:06:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1601941504 bytes 2024-09-12T16:06:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1654343680 bytes 2024-09-12T16:06:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1681895424 bytes 2024-09-12T16:06:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1699962880 bytes 2024-09-12T16:06:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1723268096 bytes 2024-09-12T16:06:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1816599552 bytes 2024-09-12T16:06:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1837062144 bytes 2024-09-12T16:06:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1882121216 bytes 2024-09-12T16:06:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1901757440 bytes 2024-09-12T16:06:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1909656576 bytes 2024-09-12T16:06:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1970955264 bytes 2024-09-12T16:06:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2023804928 bytes 2024-09-12T16:06:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2077391872 bytes 2024-09-12T16:06:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2078004224 bytes 2024-09-12T16:06:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2133998592 bytes 2024-09-12T16:06:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2134634496 bytes 2024-09-12T16:06:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2149488640 bytes 2024-09-12T16:06:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2197462016 bytes 2024-09-12T16:06:31.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.752205741176995) > weight (0.25) 2024-09-12T16:06:31.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8844442430090659) > weight (0.25) 2024-09-12T16:06:31.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2848184119879492) > weight (0.25) 2024-09-12T16:06:31.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8969884836840675) > weight (0.25) 2024-09-12T16:06:31.364 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-12T16:06:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2259720192 bytes 2024-09-12T16:06:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2290412544 bytes 2024-09-12T16:06:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2305774592 bytes 2024-09-12T16:06:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2311355392 bytes 2024-09-12T16:06:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2342305792 bytes 2024-09-12T16:06:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2367803392 bytes 2024-09-12T16:06:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2405001216 bytes 2024-09-12T16:06:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2418236416 bytes 2024-09-12T16:06:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2455153664 bytes 2024-09-12T16:06:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2459767808 bytes 2024-09-12T16:06:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2500493312 bytes 2024-09-12T16:06:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2542686208 bytes 2024-09-12T16:06:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2614269952 bytes 2024-09-12T16:06:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2701335552 bytes 2024-09-12T16:06:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:06:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:06:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:06:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:06:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_2 2024-09-12T16:06:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-12T16:06:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:06:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:06:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-12T16:06:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:06:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:06:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:06:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:06:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-12T16:06:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:06:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:06:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-12T16:06:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:06:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-12T16:06:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:06:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-12T16:06:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_2': 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1279d391c431 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1279d391c431 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:06:33 2024 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:06:33 2024 2024-09-12T16:06:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:06:33 2024 2024-09-12T16:06:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:06:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:06:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: d49cf199-0fac-4fe0-afa3-c988aa8578d1 2024-09-12T16:06:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-12T16:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:06:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:06:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:06:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 91151360 bytes 2024-09-12T16:06:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 115097600 bytes 2024-09-12T16:06:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 115710976 bytes 2024-09-12T16:06:34.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 128649216 bytes 2024-09-12T16:06:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 139404288 bytes 2024-09-12T16:06:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 215797760 bytes 2024-09-12T16:06:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 240373760 bytes 2024-09-12T16:06:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 246433792 bytes 2024-09-12T16:06:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 285339648 bytes 2024-09-12T16:06:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 290725888 bytes 2024-09-12T16:06:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 331224064 bytes 2024-09-12T16:06:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 412000256 bytes 2024-09-12T16:06:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 453228544 bytes 2024-09-12T16:06:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 496215040 bytes 2024-09-12T16:06:36.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 543421440 bytes 2024-09-12T16:06:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 567193600 bytes 2024-09-12T16:06:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 612933632 bytes 2024-09-12T16:06:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 681613312 bytes 2024-09-12T16:06:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 740506624 bytes 2024-09-12T16:06:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 794850304 bytes 2024-09-12T16:06:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 795456512 bytes 2024-09-12T16:06:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 876923904 bytes 2024-09-12T16:06:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 926340096 bytes 2024-09-12T16:06:38.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 982169600 bytes 2024-09-12T16:06:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1008834560 bytes 2024-09-12T16:06:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1042367488 bytes 2024-09-12T16:06:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1074124800 bytes 2024-09-12T16:06:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1134454784 bytes 2024-09-12T16:06:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1161619456 bytes 2024-09-12T16:06:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1184866304 bytes 2024-09-12T16:06:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1237531648 bytes 2024-09-12T16:06:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1330280448 bytes 2024-09-12T16:06:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1371124736 bytes 2024-09-12T16:06:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1451257856 bytes 2024-09-12T16:06:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1486544896 bytes 2024-09-12T16:06:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1499186176 bytes 2024-09-12T16:06:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1567405056 bytes 2024-09-12T16:06:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1574160384 bytes 2024-09-12T16:06:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1610755072 bytes 2024-09-12T16:06:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1707910144 bytes 2024-09-12T16:06:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1746026496 bytes 2024-09-12T16:06:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1785071616 bytes 2024-09-12T16:06:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1851153408 bytes 2024-09-12T16:06:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1872522240 bytes 2024-09-12T16:06:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1909361664 bytes 2024-09-12T16:06:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1918088192 bytes 2024-09-12T16:06:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1951763456 bytes 2024-09-12T16:06:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2001684480 bytes 2024-09-12T16:06:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2051497984 bytes 2024-09-12T16:06:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2080536576 bytes 2024-09-12T16:06:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2126099456 bytes 2024-09-12T16:06:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2169332736 bytes 2024-09-12T16:06:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2169939968 bytes 2024-09-12T16:06:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2182139904 bytes 2024-09-12T16:06:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2198076416 bytes 2024-09-12T16:06:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2238208000 bytes 2024-09-12T16:06:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2262529024 bytes 2024-09-12T16:06:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2305289216 bytes 2024-09-12T16:06:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2357586944 bytes 2024-09-12T16:06:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2397795328 bytes 2024-09-12T16:06:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2446769152 bytes 2024-09-12T16:06:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2472855552 bytes 2024-09-12T16:06:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2518750208 bytes 2024-09-12T16:06:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2601910272 bytes 2024-09-12T16:06:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2622468096 bytes 2024-09-12T16:06:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2658652160 bytes 2024-09-12T16:06:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2721656832 bytes 2024-09-12T16:06:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:06:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:06:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:06:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:06:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_3 2024-09-12T16:06:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-12T16:06:45.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-12T16:06:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:06:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_3': 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1285c746c16f 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1285c746c16f 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:06:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:06:45 2024 2024-09-12T16:06:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:06:45 2024 2024-09-12T16:06:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:06:45 2024 2024-09-12T16:06:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:06:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:06:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 50c9ce9d-ad98-4a52-8237-144bfb366358 2024-09-12T16:06:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-12T16:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:06:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:06:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=14 2024-09-12T16:06:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:06:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:06:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 57470976 bytes 2024-09-12T16:06:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 121056256 bytes 2024-09-12T16:06:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 194319360 bytes 2024-09-12T16:06:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 220034048 bytes 2024-09-12T16:06:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 272356352 bytes 2024-09-12T16:06:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 301143040 bytes 2024-09-12T16:06:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 330452992 bytes 2024-09-12T16:06:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 386624512 bytes 2024-09-12T16:06:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 434813952 bytes 2024-09-12T16:06:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 592966656 bytes 2024-09-12T16:06:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 638504960 bytes 2024-09-12T16:06:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 658198528 bytes 2024-09-12T16:06:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 701630464 bytes 2024-09-12T16:06:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 724910080 bytes 2024-09-12T16:06:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 788057088 bytes 2024-09-12T16:06:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 859922432 bytes 2024-09-12T16:06:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 885915648 bytes 2024-09-12T16:06:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 986847232 bytes 2024-09-12T16:06:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 1015440384 bytes 2024-09-12T16:06:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1046397952 bytes 2024-09-12T16:06:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1065314304 bytes 2024-09-12T16:06:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1097765888 bytes 2024-09-12T16:06:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1114972160 bytes 2024-09-12T16:06:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1158187008 bytes 2024-09-12T16:06:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1216822272 bytes 2024-09-12T16:06:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1227117568 bytes 2024-09-12T16:06:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1255564288 bytes 2024-09-12T16:06:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1301081088 bytes 2024-09-12T16:06:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1313499136 bytes 2024-09-12T16:06:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1337159680 bytes 2024-09-12T16:06:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1387655168 bytes 2024-09-12T16:06:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1512167424 bytes 2024-09-12T16:06:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1566526464 bytes 2024-09-12T16:06:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1582666752 bytes 2024-09-12T16:06:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1589508096 bytes 2024-09-12T16:06:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1590162432 bytes 2024-09-12T16:06:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1639653376 bytes 2024-09-12T16:06:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1702672384 bytes 2024-09-12T16:06:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1760542720 bytes 2024-09-12T16:06:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1790205952 bytes 2024-09-12T16:06:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1790809088 bytes 2024-09-12T16:06:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1822932992 bytes 2024-09-12T16:06:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1868020736 bytes 2024-09-12T16:06:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1907232768 bytes 2024-09-12T16:06:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1930476544 bytes 2024-09-12T16:06:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1958430720 bytes 2024-09-12T16:06:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2031374336 bytes 2024-09-12T16:06:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2112257024 bytes 2024-09-12T16:06:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2177855488 bytes 2024-09-12T16:06:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2214356992 bytes 2024-09-12T16:06:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2245564416 bytes 2024-09-12T16:06:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2253367296 bytes 2024-09-12T16:06:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2273685504 bytes 2024-09-12T16:06:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2319403008 bytes 2024-09-12T16:06:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2334062592 bytes 2024-09-12T16:06:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2373130240 bytes 2024-09-12T16:06:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2391601152 bytes 2024-09-12T16:06:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2463260672 bytes 2024-09-12T16:06:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2502696960 bytes 2024-09-12T16:06:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2541202432 bytes 2024-09-12T16:06:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2562642944 bytes 2024-09-12T16:06:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2606107648 bytes 2024-09-12T16:06:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2644802560 bytes 2024-09-12T16:06:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2687021056 bytes 2024-09-12T16:06:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2720933888 bytes 2024-09-12T16:06:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2784210944 bytes 2024-09-12T16:06:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2842786816 bytes 2024-09-12T16:06:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2867760128 bytes 2024-09-12T16:06:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2888133632 bytes 2024-09-12T16:06:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2963090432 bytes 2024-09-12T16:06:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 3021070336 bytes 2024-09-12T16:06:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3097274368 bytes 2024-09-12T16:06:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 3172656128 bytes 2024-09-12T16:06:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 3230574592 bytes 2024-09-12T16:06:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 3300396032 bytes 2024-09-12T16:06:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7600 IOs, 3345764352 bytes 2024-09-12T16:06:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7700 IOs, 3388406784 bytes 2024-09-12T16:06:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7800 IOs, 3480791040 bytes 2024-09-12T16:06:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7900 IOs, 3521213440 bytes 2024-09-12T16:06:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8000 IOs, 3552986112 bytes 2024-09-12T16:06:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8100 IOs, 3560342528 bytes 2024-09-12T16:06:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:06:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_4 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:06:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:06:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:06:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:06:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:06:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:06:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:06:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-12T16:07:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:07:00.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_4': 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1291d94afec3 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1291d94afec3 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:07:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:06:59 2024 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:06:59 2024 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:06:59 2024 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 2f93870e-9268-4e8c-9b38-0d6cd6a7dd81 2024-09-12T16:07:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:07:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:07:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-12T16:07:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:07:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:07:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=14 2024-09-12T16:07:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:07:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:07:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 65242112 bytes 2024-09-12T16:07:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 74654720 bytes 2024-09-12T16:07:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 121305088 bytes 2024-09-12T16:07:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 187033600 bytes 2024-09-12T16:07:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 234372096 bytes 2024-09-12T16:07:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 254585856 bytes 2024-09-12T16:07:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 281724928 bytes 2024-09-12T16:07:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 300057600 bytes 2024-09-12T16:07:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 327745536 bytes 2024-09-12T16:07:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 372204544 bytes 2024-09-12T16:07:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 430140416 bytes 2024-09-12T16:07:02.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 449710080 bytes 2024-09-12T16:07:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 491421696 bytes 2024-09-12T16:07:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 520724480 bytes 2024-09-12T16:07:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 548090880 bytes 2024-09-12T16:07:02.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 640019456 bytes 2024-09-12T16:07:03.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 691352576 bytes 2024-09-12T16:07:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 736643072 bytes 2024-09-12T16:07:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 777411584 bytes 2024-09-12T16:07:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 800728064 bytes 2024-09-12T16:07:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 860821504 bytes 2024-09-12T16:07:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 907266048 bytes 2024-09-12T16:07:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 973075456 bytes 2024-09-12T16:07:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1084259328 bytes 2024-09-12T16:07:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1117755392 bytes 2024-09-12T16:07:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1118364672 bytes 2024-09-12T16:07:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1184134144 bytes 2024-09-12T16:07:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1248155648 bytes 2024-09-12T16:07:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1280567296 bytes 2024-09-12T16:07:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1317012480 bytes 2024-09-12T16:07:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1338511360 bytes 2024-09-12T16:07:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1406104576 bytes 2024-09-12T16:07:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1429553152 bytes 2024-09-12T16:07:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1461101568 bytes 2024-09-12T16:07:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1533526016 bytes 2024-09-12T16:07:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1609789440 bytes 2024-09-12T16:07:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1643440128 bytes 2024-09-12T16:07:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1699518464 bytes 2024-09-12T16:07:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1837723648 bytes 2024-09-12T16:07:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1894602752 bytes 2024-09-12T16:07:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1917285376 bytes 2024-09-12T16:07:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1917877248 bytes 2024-09-12T16:07:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2010691584 bytes 2024-09-12T16:07:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2068126720 bytes 2024-09-12T16:07:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2123430912 bytes 2024-09-12T16:07:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2195739648 bytes 2024-09-12T16:07:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2230990848 bytes 2024-09-12T16:07:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2288073728 bytes 2024-09-12T16:07:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2315412480 bytes 2024-09-12T16:07:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2353893376 bytes 2024-09-12T16:07:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2374095872 bytes 2024-09-12T16:07:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2399685632 bytes 2024-09-12T16:07:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2447619072 bytes 2024-09-12T16:07:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2478632960 bytes 2024-09-12T16:07:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2502950912 bytes 2024-09-12T16:07:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2558179328 bytes 2024-09-12T16:07:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2638392320 bytes 2024-09-12T16:07:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2677350400 bytes 2024-09-12T16:07:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2707796992 bytes 2024-09-12T16:07:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2731643904 bytes 2024-09-12T16:07:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2763248640 bytes 2024-09-12T16:07:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2776534016 bytes 2024-09-12T16:07:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2795499520 bytes 2024-09-12T16:07:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2847963136 bytes 2024-09-12T16:07:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2929589248 bytes 2024-09-12T16:07:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2952802304 bytes 2024-09-12T16:07:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2960049152 bytes 2024-09-12T16:07:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2981344256 bytes 2024-09-12T16:07:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 3048029184 bytes 2024-09-12T16:07:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 3072961536 bytes 2024-09-12T16:07:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 3095142400 bytes 2024-09-12T16:07:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3134880768 bytes 2024-09-12T16:07:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 3149152256 bytes 2024-09-12T16:07:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 3214252032 bytes 2024-09-12T16:07:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 3287530496 bytes 2024-09-12T16:07:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7600 IOs, 3357715456 bytes 2024-09-12T16:07:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7700 IOs, 3473468416 bytes 2024-09-12T16:07:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7800 IOs, 3477628928 bytes 2024-09-12T16:07:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7900 IOs, 3559395328 bytes 2024-09-12T16:07:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8000 IOs, 3617744896 bytes 2024-09-12T16:07:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8100 IOs, 3653419008 bytes 2024-09-12T16:07:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:07:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:07:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:07:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_5 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:07:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-12T16:07:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:07:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:07:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-12T16:07:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:07:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-12T16:07:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:07:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-12T16:07:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_5': 2024-09-12T16:07:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:07:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:07:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:07:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 129cecc6fe12 2024-09-12T16:07:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.129cecc6fe12 2024-09-12T16:07:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:14 2024 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:14 2024 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:14 2024 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 4e25a06d-cee5-4e43-8e00-5e4bc8c2f14a 2024-09-12T16:07:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:07:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:07:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-12T16:07:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:07:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:07:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=7 2024-09-12T16:07:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:07:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:07:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 106946560 bytes 2024-09-12T16:07:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 165048320 bytes 2024-09-12T16:07:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 192401408 bytes 2024-09-12T16:07:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 245882880 bytes 2024-09-12T16:07:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 269462528 bytes 2024-09-12T16:07:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 303538176 bytes 2024-09-12T16:07:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 360843264 bytes 2024-09-12T16:07:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 381644800 bytes 2024-09-12T16:07:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 425402368 bytes 2024-09-12T16:07:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 426007552 bytes 2024-09-12T16:07:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 464702464 bytes 2024-09-12T16:07:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 500753408 bytes 2024-09-12T16:07:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 561482752 bytes 2024-09-12T16:07:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 615784448 bytes 2024-09-12T16:07:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 724376576 bytes 2024-09-12T16:07:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 737648640 bytes 2024-09-12T16:07:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 796238848 bytes 2024-09-12T16:07:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 845227008 bytes 2024-09-12T16:07:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 879386624 bytes 2024-09-12T16:07:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 956656640 bytes 2024-09-12T16:07:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 967254016 bytes 2024-09-12T16:07:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 997552128 bytes 2024-09-12T16:07:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1082726400 bytes 2024-09-12T16:07:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1152910336 bytes 2024-09-12T16:07:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1171427328 bytes 2024-09-12T16:07:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1243999232 bytes 2024-09-12T16:07:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1287961600 bytes 2024-09-12T16:07:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1343160320 bytes 2024-09-12T16:07:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1366588416 bytes 2024-09-12T16:07:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1421227008 bytes 2024-09-12T16:07:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1486588928 bytes 2024-09-12T16:07:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1528561664 bytes 2024-09-12T16:07:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1590068224 bytes 2024-09-12T16:07:21.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1633629184 bytes 2024-09-12T16:07:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1700967424 bytes 2024-09-12T16:07:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1706181632 bytes 2024-09-12T16:07:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1738915840 bytes 2024-09-12T16:07:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1739540480 bytes 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_6 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:07:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:07:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:07:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:07:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-12T16:07:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:07:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:07:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-12T16:07:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:07:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-12T16:07:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:07:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-12T16:07:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_6': 2024-09-12T16:07:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:07:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 12a887110535 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.12a887110535 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:22 2024 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:22 2024 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:22 2024 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 7116ac0d-b88f-466e-8403-60e8e5d9a276 2024-09-12T16:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:07:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:07:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-12T16:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:07:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:07:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:07:23.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:07:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 87061504 bytes 2024-09-12T16:07:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 103441408 bytes 2024-09-12T16:07:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 150388736 bytes 2024-09-12T16:07:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 217367552 bytes 2024-09-12T16:07:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 256821248 bytes 2024-09-12T16:07:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 284844032 bytes 2024-09-12T16:07:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 340930560 bytes 2024-09-12T16:07:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 357720064 bytes 2024-09-12T16:07:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 378582016 bytes 2024-09-12T16:07:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 416055296 bytes 2024-09-12T16:07:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 471239680 bytes 2024-09-12T16:07:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 496549888 bytes 2024-09-12T16:07:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 644715520 bytes 2024-09-12T16:07:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 686155776 bytes 2024-09-12T16:07:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 723394560 bytes 2024-09-12T16:07:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 728144896 bytes 2024-09-12T16:07:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 774384640 bytes 2024-09-12T16:07:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 823513088 bytes 2024-09-12T16:07:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 872949760 bytes 2024-09-12T16:07:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 905790464 bytes 2024-09-12T16:07:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 935842816 bytes 2024-09-12T16:07:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 964847616 bytes 2024-09-12T16:07:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 986385408 bytes 2024-09-12T16:07:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1043493888 bytes 2024-09-12T16:07:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1131427840 bytes 2024-09-12T16:07:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1153894400 bytes 2024-09-12T16:07:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1219378176 bytes 2024-09-12T16:07:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1248891904 bytes 2024-09-12T16:07:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1301884928 bytes 2024-09-12T16:07:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1318992896 bytes 2024-09-12T16:07:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1319794688 bytes 2024-09-12T16:07:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1346988032 bytes 2024-09-12T16:07:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1382514688 bytes 2024-09-12T16:07:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1438930944 bytes 2024-09-12T16:07:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1477096448 bytes 2024-09-12T16:07:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1523943424 bytes 2024-09-12T16:07:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1548689408 bytes 2024-09-12T16:07:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1589190656 bytes 2024-09-12T16:07:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1619656704 bytes 2024-09-12T16:07:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1650233344 bytes 2024-09-12T16:07:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1739544576 bytes 2024-09-12T16:07:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1758776320 bytes 2024-09-12T16:07:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1840592896 bytes 2024-09-12T16:07:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1850637312 bytes 2024-09-12T16:07:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1878375424 bytes 2024-09-12T16:07:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1887270912 bytes 2024-09-12T16:07:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1899867136 bytes 2024-09-12T16:07:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1907802112 bytes 2024-09-12T16:07:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1989758976 bytes 2024-09-12T16:07:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2016021504 bytes 2024-09-12T16:07:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2087032832 bytes 2024-09-12T16:07:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2162941952 bytes 2024-09-12T16:07:32.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2186644480 bytes 2024-09-12T16:07:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2269833216 bytes 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_7 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:07:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:07:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:07:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:07:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-12T16:07:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:07:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:07:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-12T16:07:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:07:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-12T16:07:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:07:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-12T16:07:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_7': 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 12b482e1f992 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.12b482e1f992 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:32 2024 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:32 2024 2024-09-12T16:07:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:32 2024 2024-09-12T16:07:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:07:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:07:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 9ed94e20-8486-411e-a66b-35ec4ef5bf45 2024-09-12T16:07:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:07:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:07:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-12T16:07:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:07:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:07:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:07:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:07:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:07:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 55042048 bytes 2024-09-12T16:07:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 55677952 bytes 2024-09-12T16:07:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 77148160 bytes 2024-09-12T16:07:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 146089984 bytes 2024-09-12T16:07:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 158126080 bytes 2024-09-12T16:07:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 192053248 bytes 2024-09-12T16:07:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 276924416 bytes 2024-09-12T16:07:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 357818368 bytes 2024-09-12T16:07:35.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 390054912 bytes 2024-09-12T16:07:35.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 399203328 bytes 2024-09-12T16:07:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 439183360 bytes 2024-09-12T16:07:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 539675648 bytes 2024-09-12T16:07:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 598449152 bytes 2024-09-12T16:07:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 693449728 bytes 2024-09-12T16:07:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 736545792 bytes 2024-09-12T16:07:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 757662720 bytes 2024-09-12T16:07:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 820448256 bytes 2024-09-12T16:07:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 868659200 bytes 2024-09-12T16:07:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 943755264 bytes 2024-09-12T16:07:37.365 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7749389270706183) > weight (0.25) 2024-09-12T16:07:37.365 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5883948172204698) > weight (0.25) 2024-09-12T16:07:37.365 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9198562914461342) > weight (0.25) 2024-09-12T16:07:37.365 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-12T16:07:37.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-12T16:07:37.366 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-12T16:07:37.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:07:37.366+0000 7fe3c18c6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 40381) UID: 1000 2024-09-12T16:07:37.378 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.41440782399573506) > weight (0.25) 2024-09-12T16:07:37.378 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6221553632048091) > weight (0.25) 2024-09-12T16:07:37.378 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.38142394290933423) > weight (0.25) 2024-09-12T16:07:37.378 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.40951269649002286) > weight (0.25) 2024-09-12T16:07:37.379 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-12T16:07:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1046515712 bytes 2024-09-12T16:07:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1075738624 bytes 2024-09-12T16:07:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1209570304 bytes 2024-09-12T16:07:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1219169280 bytes 2024-09-12T16:07:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1276483584 bytes 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_8 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:07:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:07:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:07:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:07:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-12T16:07:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:07:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-12T16:07:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_8': 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 12c077be5bb3 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.12c077be5bb3 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:38 2024 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:38 2024 2024-09-12T16:07:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:38 2024 2024-09-12T16:07:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:07:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:07:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 7246287d-3dc0-422d-9d93-915c9d37781e 2024-09-12T16:07:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:07:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:07:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-12T16:07:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:07:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:07:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=10 2024-09-12T16:07:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:07:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:07:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 70981632 bytes 2024-09-12T16:07:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 123872256 bytes 2024-09-12T16:07:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 137849856 bytes 2024-09-12T16:07:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 177340416 bytes 2024-09-12T16:07:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 235141120 bytes 2024-09-12T16:07:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 275126272 bytes 2024-09-12T16:07:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 318181376 bytes 2024-09-12T16:07:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 348971008 bytes 2024-09-12T16:07:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 437298176 bytes 2024-09-12T16:07:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 500136960 bytes 2024-09-12T16:07:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 613666816 bytes 2024-09-12T16:07:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 642435072 bytes 2024-09-12T16:07:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 685100032 bytes 2024-09-12T16:07:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 707929088 bytes 2024-09-12T16:07:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 771666944 bytes 2024-09-12T16:07:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 786205696 bytes 2024-09-12T16:07:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 823836672 bytes 2024-09-12T16:07:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 854806528 bytes 2024-09-12T16:07:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 871597056 bytes 2024-09-12T16:07:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 947554304 bytes 2024-09-12T16:07:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 984884224 bytes 2024-09-12T16:07:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1006441472 bytes 2024-09-12T16:07:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1049528320 bytes 2024-09-12T16:07:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1079738368 bytes 2024-09-12T16:07:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1094279168 bytes 2024-09-12T16:07:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1152162816 bytes 2024-09-12T16:07:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1181833216 bytes 2024-09-12T16:07:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1235995648 bytes 2024-09-12T16:07:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1272574976 bytes 2024-09-12T16:07:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1326883840 bytes 2024-09-12T16:07:44.367 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-12T16:07:44.367 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:07:44.367 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-12T16:07:44.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-12T16:07:44.368 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-12T16:07:44.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-12T16:07:44.370 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-12T16:07:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1394308096 bytes 2024-09-12T16:07:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1422136320 bytes 2024-09-12T16:07:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1496104960 bytes 2024-09-12T16:07:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1509476352 bytes 2024-09-12T16:07:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1534405632 bytes 2024-09-12T16:07:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1615016960 bytes 2024-09-12T16:07:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1674574848 bytes 2024-09-12T16:07:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1691890688 bytes 2024-09-12T16:07:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1712508928 bytes 2024-09-12T16:07:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1762270208 bytes 2024-09-12T16:07:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1789151232 bytes 2024-09-12T16:07:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1830026240 bytes 2024-09-12T16:07:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1880539136 bytes 2024-09-12T16:07:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1896986624 bytes 2024-09-12T16:07:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1960401920 bytes 2024-09-12T16:07:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1973166080 bytes 2024-09-12T16:07:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2012514304 bytes 2024-09-12T16:07:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2053134336 bytes 2024-09-12T16:07:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2065844224 bytes 2024-09-12T16:07:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2122731520 bytes 2024-09-12T16:07:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2153735168 bytes 2024-09-12T16:07:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2186057728 bytes 2024-09-12T16:07:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2194587648 bytes 2024-09-12T16:07:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2242460672 bytes 2024-09-12T16:07:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2335119360 bytes 2024-09-12T16:07:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2338845696 bytes 2024-09-12T16:07:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2387763200 bytes 2024-09-12T16:07:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2422679552 bytes 2024-09-12T16:07:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_9 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:07:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:07:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:07:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-12T16:07:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:07:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-12T16:07:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:07:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_9': 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 12d270e87510 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.12d270e87510 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:07:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:07:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:48 2024 2024-09-12T16:07:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:48 2024 2024-09-12T16:07:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:48 2024 2024-09-12T16:07:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:07:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:07:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: f574749e-1611-4718-9f27-0329fe616606 2024-09-12T16:07:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:07:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:07:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-12T16:07:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:07:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:07:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:07:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:07:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:07:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:07:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:07:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 77045760 bytes 2024-09-12T16:07:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 159283200 bytes 2024-09-12T16:07:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 173464576 bytes 2024-09-12T16:07:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 206181376 bytes 2024-09-12T16:07:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 264179712 bytes 2024-09-12T16:07:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 296433664 bytes 2024-09-12T16:07:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 297042944 bytes 2024-09-12T16:07:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 382601216 bytes 2024-09-12T16:07:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 383211520 bytes 2024-09-12T16:07:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 401141760 bytes 2024-09-12T16:07:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 416345088 bytes 2024-09-12T16:07:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 494597120 bytes 2024-09-12T16:07:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 523541504 bytes 2024-09-12T16:07:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 524142592 bytes 2024-09-12T16:07:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 638161920 bytes 2024-09-12T16:07:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 722288640 bytes 2024-09-12T16:07:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 763140096 bytes 2024-09-12T16:07:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 818669568 bytes 2024-09-12T16:07:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 823199744 bytes 2024-09-12T16:07:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 857246720 bytes 2024-09-12T16:07:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 881187840 bytes 2024-09-12T16:07:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 959477760 bytes 2024-09-12T16:07:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1000585216 bytes 2024-09-12T16:07:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1053979648 bytes 2024-09-12T16:07:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1094731776 bytes 2024-09-12T16:07:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1113004032 bytes 2024-09-12T16:07:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1154288640 bytes 2024-09-12T16:07:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1198930944 bytes 2024-09-12T16:07:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1279235072 bytes 2024-09-12T16:07:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1341711360 bytes 2024-09-12T16:07:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1376620544 bytes 2024-09-12T16:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1460541440 bytes 2024-09-12T16:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1523079168 bytes 2024-09-12T16:07:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1574115328 bytes 2024-09-12T16:07:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1641025536 bytes 2024-09-12T16:07:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1660347392 bytes 2024-09-12T16:07:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1704085504 bytes 2024-09-12T16:07:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1805211648 bytes 2024-09-12T16:07:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1813179392 bytes 2024-09-12T16:07:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1845925888 bytes 2024-09-12T16:07:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1923021824 bytes 2024-09-12T16:07:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1975915520 bytes 2024-09-12T16:07:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1978337280 bytes 2024-09-12T16:07:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1985444864 bytes 2024-09-12T16:07:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2030862336 bytes 2024-09-12T16:07:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2054096896 bytes 2024-09-12T16:07:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2095540224 bytes 2024-09-12T16:07:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2145959936 bytes 2024-09-12T16:07:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2154338304 bytes 2024-09-12T16:07:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2234283008 bytes 2024-09-12T16:07:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2301300736 bytes 2024-09-12T16:07:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2330408960 bytes 2024-09-12T16:07:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2404039680 bytes 2024-09-12T16:07:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2419749888 bytes 2024-09-12T16:07:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2470100992 bytes 2024-09-12T16:07:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2499083264 bytes 2024-09-12T16:07:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2547900416 bytes 2024-09-12T16:07:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2595805184 bytes 2024-09-12T16:07:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2596426752 bytes 2024-09-12T16:07:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2669466624 bytes 2024-09-12T16:07:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2670080000 bytes 2024-09-12T16:08:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2704653312 bytes 2024-09-12T16:08:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2728396800 bytes 2024-09-12T16:08:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:08:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:08:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:08:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:08:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_10 2024-09-12T16:08:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-12T16:08:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:08:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:08:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-12T16:08:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-12T16:08:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:08:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:08:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-12T16:08:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:08:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-12T16:08:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:08:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_10': 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 12de2d842122 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.12de2d842122 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:08:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:08:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:00 2024 2024-09-12T16:08:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:00 2024 2024-09-12T16:08:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:00 2024 2024-09-12T16:08:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:08:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:08:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c558b2dc-a46e-4a16-bc7f-98af17a10d37 2024-09-12T16:08:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:08:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:08:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-12T16:08:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:08:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:08:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=7 2024-09-12T16:08:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:08:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:08:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 7606272 bytes 2024-09-12T16:08:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 55433216 bytes 2024-09-12T16:08:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 85463040 bytes 2024-09-12T16:08:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 130650112 bytes 2024-09-12T16:08:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 149001216 bytes 2024-09-12T16:08:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 149627904 bytes 2024-09-12T16:08:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 174167040 bytes 2024-09-12T16:08:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 197607424 bytes 2024-09-12T16:08:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 224095232 bytes 2024-09-12T16:08:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 273058816 bytes 2024-09-12T16:08:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 285346816 bytes 2024-09-12T16:08:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 402821120 bytes 2024-09-12T16:08:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 525802496 bytes 2024-09-12T16:08:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 526417920 bytes 2024-09-12T16:08:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 581074944 bytes 2024-09-12T16:08:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 709362688 bytes 2024-09-12T16:08:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 719736832 bytes 2024-09-12T16:08:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 747249664 bytes 2024-09-12T16:08:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 825843712 bytes 2024-09-12T16:08:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 888977408 bytes 2024-09-12T16:08:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 970692608 bytes 2024-09-12T16:08:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1003566080 bytes 2024-09-12T16:08:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1020692480 bytes 2024-09-12T16:08:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1103747072 bytes 2024-09-12T16:08:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1144377344 bytes 2024-09-12T16:08:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1186014208 bytes 2024-09-12T16:08:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1222028288 bytes 2024-09-12T16:08:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1259831296 bytes 2024-09-12T16:08:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1298370560 bytes 2024-09-12T16:08:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1360518144 bytes 2024-09-12T16:08:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1433928704 bytes 2024-09-12T16:08:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1490190336 bytes 2024-09-12T16:08:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1517025280 bytes 2024-09-12T16:08:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1542302720 bytes 2024-09-12T16:08:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1595401216 bytes 2024-09-12T16:08:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1670267904 bytes 2024-09-12T16:08:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1746983936 bytes 2024-09-12T16:08:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1772108800 bytes 2024-09-12T16:08:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1790504960 bytes 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_11 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:08:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:08:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:08:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:08:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:08:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-12T16:08:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:08:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_11': 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 12eae0cbd289 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.12eae0cbd289 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:08:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:08 2024 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:08 2024 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:08 2024 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c7fac33d-e73d-4d4b-8daf-469b5cf5e4e4 2024-09-12T16:08:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:08:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:08:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-12T16:08:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:08:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:08:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:08:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:08:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:08:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 76697600 bytes 2024-09-12T16:08:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 130029568 bytes 2024-09-12T16:08:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 134946816 bytes 2024-09-12T16:08:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 213584896 bytes 2024-09-12T16:08:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 240014336 bytes 2024-09-12T16:08:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 285308928 bytes 2024-09-12T16:08:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 316635136 bytes 2024-09-12T16:08:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 343710720 bytes 2024-09-12T16:08:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 409127936 bytes 2024-09-12T16:08:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 469479424 bytes 2024-09-12T16:08:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 507509760 bytes 2024-09-12T16:08:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 547785728 bytes 2024-09-12T16:08:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 586125312 bytes 2024-09-12T16:08:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 623351808 bytes 2024-09-12T16:08:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 691387392 bytes 2024-09-12T16:08:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 699285504 bytes 2024-09-12T16:08:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 783279104 bytes 2024-09-12T16:08:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 852084736 bytes 2024-09-12T16:08:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 852700160 bytes 2024-09-12T16:08:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 894151680 bytes 2024-09-12T16:08:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 936201216 bytes 2024-09-12T16:08:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 959313920 bytes 2024-09-12T16:08:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1009418240 bytes 2024-09-12T16:08:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1044410368 bytes 2024-09-12T16:08:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1126359040 bytes 2024-09-12T16:08:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1150810112 bytes 2024-09-12T16:08:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1155239936 bytes 2024-09-12T16:08:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1184338944 bytes 2024-09-12T16:08:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1274645504 bytes 2024-09-12T16:08:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1292883968 bytes 2024-09-12T16:08:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1329628160 bytes 2024-09-12T16:08:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1380768768 bytes 2024-09-12T16:08:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1404740608 bytes 2024-09-12T16:08:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1405356032 bytes 2024-09-12T16:08:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1405964288 bytes 2024-09-12T16:08:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1433165824 bytes 2024-09-12T16:08:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1467784192 bytes 2024-09-12T16:08:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1530013696 bytes 2024-09-12T16:08:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1574155264 bytes 2024-09-12T16:08:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1649787904 bytes 2024-09-12T16:08:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1670637568 bytes 2024-09-12T16:08:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1718874112 bytes 2024-09-12T16:08:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1775128576 bytes 2024-09-12T16:08:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1832957952 bytes 2024-09-12T16:08:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1852554240 bytes 2024-09-12T16:08:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1882129408 bytes 2024-09-12T16:08:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1937820672 bytes 2024-09-12T16:08:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1958873088 bytes 2024-09-12T16:08:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1959483392 bytes 2024-09-12T16:08:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2007405568 bytes 2024-09-12T16:08:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2102512640 bytes 2024-09-12T16:08:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2126631936 bytes 2024-09-12T16:08:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2139331584 bytes 2024-09-12T16:08:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2159380480 bytes 2024-09-12T16:08:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2171222016 bytes 2024-09-12T16:08:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2209145856 bytes 2024-09-12T16:08:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2255338496 bytes 2024-09-12T16:08:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2365652992 bytes 2024-09-12T16:08:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2455290880 bytes 2024-09-12T16:08:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2515914752 bytes 2024-09-12T16:08:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2551131136 bytes 2024-09-12T16:08:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2557459456 bytes 2024-09-12T16:08:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2607751168 bytes 2024-09-12T16:08:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2652263424 bytes 2024-09-12T16:08:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2672614400 bytes 2024-09-12T16:08:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:08:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_12 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:08:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:08:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:08:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:08:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-09-12T16:08:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-12T16:08:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:08:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-12T16:08:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_12': 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 12f6fdb43c55 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.12f6fdb43c55 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:20 2024 2024-09-12T16:08:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:20 2024 2024-09-12T16:08:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:20 2024 2024-09-12T16:08:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:08:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:08:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c928224c-98a5-45be-8f56-7a1ae44aa427 2024-09-12T16:08:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:08:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:08:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-12T16:08:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:08:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:08:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T16:08:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:08:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:08:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 41912320 bytes 2024-09-12T16:08:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 72132608 bytes 2024-09-12T16:08:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 108665856 bytes 2024-09-12T16:08:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 156917760 bytes 2024-09-12T16:08:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 206725120 bytes 2024-09-12T16:08:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 245445632 bytes 2024-09-12T16:08:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 294861824 bytes 2024-09-12T16:08:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 305559552 bytes 2024-09-12T16:08:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 311047168 bytes 2024-09-12T16:08:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 354653184 bytes 2024-09-12T16:08:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 379635712 bytes 2024-09-12T16:08:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 452858880 bytes 2024-09-12T16:08:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 503930880 bytes 2024-09-12T16:08:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 539092992 bytes 2024-09-12T16:08:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 634509312 bytes 2024-09-12T16:08:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 669938688 bytes 2024-09-12T16:08:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 686944256 bytes 2024-09-12T16:08:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 689994752 bytes 2024-09-12T16:08:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 752957440 bytes 2024-09-12T16:08:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 757053440 bytes 2024-09-12T16:08:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 791972864 bytes 2024-09-12T16:08:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 816975872 bytes 2024-09-12T16:08:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 843160576 bytes 2024-09-12T16:08:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 883303424 bytes 2024-09-12T16:08:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 985109504 bytes 2024-09-12T16:08:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1042759680 bytes 2024-09-12T16:08:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1146123264 bytes 2024-09-12T16:08:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1219100672 bytes 2024-09-12T16:08:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1221213184 bytes 2024-09-12T16:08:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1240417280 bytes 2024-09-12T16:08:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1241012224 bytes 2024-09-12T16:08:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1269813248 bytes 2024-09-12T16:08:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1325057024 bytes 2024-09-12T16:08:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1387163648 bytes 2024-09-12T16:08:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1448765440 bytes 2024-09-12T16:08:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1449377792 bytes 2024-09-12T16:08:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1526484992 bytes 2024-09-12T16:08:27.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1600431104 bytes 2024-09-12T16:08:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1663593472 bytes 2024-09-12T16:08:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1699409920 bytes 2024-09-12T16:08:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1740401664 bytes 2024-09-12T16:08:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1800759296 bytes 2024-09-12T16:08:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1879735296 bytes 2024-09-12T16:08:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1921255424 bytes 2024-09-12T16:08:29.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2009578496 bytes 2024-09-12T16:08:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2088932352 bytes 2024-09-12T16:08:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2137397248 bytes 2024-09-12T16:08:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2152781824 bytes 2024-09-12T16:08:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2221709312 bytes 2024-09-12T16:08:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2294767616 bytes 2024-09-12T16:08:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2349501440 bytes 2024-09-12T16:08:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2393989120 bytes 2024-09-12T16:08:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2470444032 bytes 2024-09-12T16:08:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2482430976 bytes 2024-09-12T16:08:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2491640832 bytes 2024-09-12T16:08:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2594290688 bytes 2024-09-12T16:08:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2707660800 bytes 2024-09-12T16:08:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2743660544 bytes 2024-09-12T16:08:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2799293440 bytes 2024-09-12T16:08:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2849511424 bytes 2024-09-12T16:08:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2883093504 bytes 2024-09-12T16:08:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2894948352 bytes 2024-09-12T16:08:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2936544256 bytes 2024-09-12T16:08:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2962529280 bytes 2024-09-12T16:08:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2986091520 bytes 2024-09-12T16:08:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 3034561536 bytes 2024-09-12T16:08:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_13 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:08:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:08:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:08:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-09-12T16:08:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-12T16:08:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:08:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-12T16:08:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_13': 2024-09-12T16:08:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:08:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:08:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:08:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1302ebe1a43 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1302ebe1a43 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:33 2024 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:33 2024 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:33 2024 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 02a2eb1f-5638-48d4-814e-2b3a49935a89 2024-09-12T16:08:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:08:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:08:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-12T16:08:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:08:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:08:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:08:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:08:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:08:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 58015744 bytes 2024-09-12T16:08:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 116929536 bytes 2024-09-12T16:08:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 150598656 bytes 2024-09-12T16:08:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 204088320 bytes 2024-09-12T16:08:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 259008512 bytes 2024-09-12T16:08:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 307121152 bytes 2024-09-12T16:08:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 333007872 bytes 2024-09-12T16:08:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 394292224 bytes 2024-09-12T16:08:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 484236288 bytes 2024-09-12T16:08:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 527872000 bytes 2024-09-12T16:08:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 606298112 bytes 2024-09-12T16:08:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 669173760 bytes 2024-09-12T16:08:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 669799424 bytes 2024-09-12T16:08:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 692554752 bytes 2024-09-12T16:08:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 708870144 bytes 2024-09-12T16:08:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 787929088 bytes 2024-09-12T16:08:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 829548544 bytes 2024-09-12T16:08:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 899816448 bytes 2024-09-12T16:08:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 923025408 bytes 2024-09-12T16:08:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1007400960 bytes 2024-09-12T16:08:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1101821952 bytes 2024-09-12T16:08:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1123303424 bytes 2024-09-12T16:08:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1158890496 bytes 2024-09-12T16:08:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1199773696 bytes 2024-09-12T16:08:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1225197568 bytes 2024-09-12T16:08:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1346872320 bytes 2024-09-12T16:08:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1415485440 bytes 2024-09-12T16:08:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1475332096 bytes 2024-09-12T16:08:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1505983488 bytes 2024-09-12T16:08:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1566888960 bytes 2024-09-12T16:08:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1667147776 bytes 2024-09-12T16:08:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1718013952 bytes 2024-09-12T16:08:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1743367168 bytes 2024-09-12T16:08:40.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1786278912 bytes 2024-09-12T16:08:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1845838848 bytes 2024-09-12T16:08:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1873329152 bytes 2024-09-12T16:08:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1944470528 bytes 2024-09-12T16:08:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1958136832 bytes 2024-09-12T16:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 2011066368 bytes 2024-09-12T16:08:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 2049229824 bytes 2024-09-12T16:08:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 2077495296 bytes 2024-09-12T16:08:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 2142700544 bytes 2024-09-12T16:08:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2157454336 bytes 2024-09-12T16:08:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2230637568 bytes 2024-09-12T16:08:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2254546944 bytes 2024-09-12T16:08:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2298260480 bytes 2024-09-12T16:08:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2302002176 bytes 2024-09-12T16:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2360722432 bytes 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_14 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-12T16:08:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-09-12T16:08:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:08:43.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-12T16:08:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:08:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_14': 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 130e82d5951d 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.130e82d5951d 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:08:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:43 2024 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:43 2024 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:43 2024 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 4d0d2df1-b0b8-4abb-b611-0cc3a70369d4 2024-09-12T16:08:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:08:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:08:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-12T16:08:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:08:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:08:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=7 2024-09-12T16:08:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:08:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:08:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 48603136 bytes 2024-09-12T16:08:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 67347456 bytes 2024-09-12T16:08:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 106396672 bytes 2024-09-12T16:08:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 144256000 bytes 2024-09-12T16:08:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 152136704 bytes 2024-09-12T16:08:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 211365888 bytes 2024-09-12T16:08:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 283018240 bytes 2024-09-12T16:08:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 325358592 bytes 2024-09-12T16:08:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 358391808 bytes 2024-09-12T16:08:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 390180864 bytes 2024-09-12T16:08:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 418695168 bytes 2024-09-12T16:08:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 471233536 bytes 2024-09-12T16:08:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 513532928 bytes 2024-09-12T16:08:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 544876544 bytes 2024-09-12T16:08:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 561653760 bytes 2024-09-12T16:08:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 597932032 bytes 2024-09-12T16:08:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 635209728 bytes 2024-09-12T16:08:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 668776448 bytes 2024-09-12T16:08:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 704056320 bytes 2024-09-12T16:08:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 772538368 bytes 2024-09-12T16:08:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 818522112 bytes 2024-09-12T16:08:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 888393728 bytes 2024-09-12T16:08:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 931203072 bytes 2024-09-12T16:08:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 934142976 bytes 2024-09-12T16:08:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 976146432 bytes 2024-09-12T16:08:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1006240768 bytes 2024-09-12T16:08:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1019969536 bytes 2024-09-12T16:08:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1073034240 bytes 2024-09-12T16:08:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1099091968 bytes 2024-09-12T16:08:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1133296640 bytes 2024-09-12T16:08:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1193442304 bytes 2024-09-12T16:08:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1232494592 bytes 2024-09-12T16:08:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1263718400 bytes 2024-09-12T16:08:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1363471360 bytes 2024-09-12T16:08:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1390768128 bytes 2024-09-12T16:08:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1416348672 bytes 2024-09-12T16:08:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1460357120 bytes 2024-09-12T16:08:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1513341952 bytes 2024-09-12T16:08:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1550237696 bytes 2024-09-12T16:08:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1574545408 bytes 2024-09-12T16:08:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1607163904 bytes 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_15 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-12T16:08:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:08:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:08:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:08:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:08:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:08:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-09-12T16:08:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:08:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:08:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:08:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-12T16:08:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:08:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:08:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-12T16:08:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:08:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-12T16:08:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_15': 2024-09-12T16:08:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 131aac1af99d 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.131aac1af99d 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:51 2024 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:51 2024 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:51 2024 2024-09-12T16:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:08:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:08:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 1961696a-f14e-44a0-a222-ad6e28c429fe 2024-09-12T16:08:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:08:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:08:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-12T16:08:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:08:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:08:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:08:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:08:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=14 2024-09-12T16:08:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:08:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:08:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 78514176 bytes 2024-09-12T16:08:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 115023872 bytes 2024-09-12T16:08:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 115653632 bytes 2024-09-12T16:08:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 151393280 bytes 2024-09-12T16:08:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 199755776 bytes 2024-09-12T16:08:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 242951168 bytes 2024-09-12T16:08:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 273353728 bytes 2024-09-12T16:08:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 273994752 bytes 2024-09-12T16:08:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 300544000 bytes 2024-09-12T16:08:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 334555136 bytes 2024-09-12T16:08:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 374584320 bytes 2024-09-12T16:08:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 383765504 bytes 2024-09-12T16:08:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 395853824 bytes 2024-09-12T16:08:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 423608320 bytes 2024-09-12T16:08:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 435514368 bytes 2024-09-12T16:08:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 473512960 bytes 2024-09-12T16:08:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 528680960 bytes 2024-09-12T16:08:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 538609664 bytes 2024-09-12T16:08:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 548044800 bytes 2024-09-12T16:08:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 594847744 bytes 2024-09-12T16:08:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 639212544 bytes 2024-09-12T16:08:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 657324032 bytes 2024-09-12T16:08:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 721235968 bytes 2024-09-12T16:08:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 745600000 bytes 2024-09-12T16:08:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 769135616 bytes 2024-09-12T16:08:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 780158976 bytes 2024-09-12T16:08:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 793195520 bytes 2024-09-12T16:08:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 820952064 bytes 2024-09-12T16:08:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 821558272 bytes 2024-09-12T16:08:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 859631616 bytes 2024-09-12T16:08:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 870051840 bytes 2024-09-12T16:08:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 908393472 bytes 2024-09-12T16:08:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 962141184 bytes 2024-09-12T16:08:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 993052672 bytes 2024-09-12T16:08:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1018662912 bytes 2024-09-12T16:08:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1076340736 bytes 2024-09-12T16:08:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1125974016 bytes 2024-09-12T16:08:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1156792320 bytes 2024-09-12T16:08:58.379 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.758225037010539) > weight (0.25) 2024-09-12T16:08:58.379 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-12T16:08:58.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-12T16:08:58.380 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-12T16:08:58.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:08:58.380+0000 7f514d9f6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39792) UID: 1000 2024-09-12T16:08:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1221006336 bytes 2024-09-12T16:08:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1240531968 bytes 2024-09-12T16:08:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1256700928 bytes 2024-09-12T16:08:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1261103104 bytes 2024-09-12T16:08:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1268336640 bytes 2024-09-12T16:08:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1365757952 bytes 2024-09-12T16:08:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1366389760 bytes 2024-09-12T16:08:59.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1424076800 bytes 2024-09-12T16:08:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1482507264 bytes 2024-09-12T16:08:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1483124736 bytes 2024-09-12T16:09:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1548296192 bytes 2024-09-12T16:09:00.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1577848832 bytes 2024-09-12T16:09:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 1632471040 bytes 2024-09-12T16:09:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 1667807232 bytes 2024-09-12T16:09:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 1674423296 bytes 2024-09-12T16:09:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 1704388608 bytes 2024-09-12T16:09:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 1729860608 bytes 2024-09-12T16:09:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 1800598528 bytes 2024-09-12T16:09:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 1834971136 bytes 2024-09-12T16:09:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 1855770624 bytes 2024-09-12T16:09:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 1894359040 bytes 2024-09-12T16:09:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 1960854528 bytes 2024-09-12T16:09:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2017055744 bytes 2024-09-12T16:09:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2032353280 bytes 2024-09-12T16:09:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2134298624 bytes 2024-09-12T16:09:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2185777152 bytes 2024-09-12T16:09:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2204488704 bytes 2024-09-12T16:09:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2208076800 bytes 2024-09-12T16:09:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2219845632 bytes 2024-09-12T16:09:03.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2268306432 bytes 2024-09-12T16:09:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2268906496 bytes 2024-09-12T16:09:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2325166080 bytes 2024-09-12T16:09:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 2393338880 bytes 2024-09-12T16:09:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 2444365824 bytes 2024-09-12T16:09:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 2475298816 bytes 2024-09-12T16:09:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 2500141056 bytes 2024-09-12T16:09:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 2567724032 bytes 2024-09-12T16:09:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7600 IOs, 2642996224 bytes 2024-09-12T16:09:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7700 IOs, 2733357056 bytes 2024-09-12T16:09:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7800 IOs, 2842801152 bytes 2024-09-12T16:09:05.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7900 IOs, 2947586048 bytes 2024-09-12T16:09:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8000 IOs, 2948178944 bytes 2024-09-12T16:09:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8100 IOs, 3019847680 bytes 2024-09-12T16:09:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8200 IOs, 3050823680 bytes 2024-09-12T16:09:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8300 IOs, 3078056960 bytes 2024-09-12T16:09:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8400 IOs, 3093700608 bytes 2024-09-12T16:09:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:8500 IOs, 3099191296 bytes 2024-09-12T16:09:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:09:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_16 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:09:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:09:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-09-12T16:09:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:09:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:09:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-12T16:09:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:09:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-12T16:09:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:09:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-12T16:09:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_16': 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1326e41b8dfe 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1326e41b8dfe 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:06 2024 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:06 2024 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:06 2024 2024-09-12T16:09:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:09:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:09:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 47d38d69-792b-405f-a65b-101acd193fb9 2024-09-12T16:09:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:09:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:09:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-12T16:09:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:09:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:09:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=13 2024-09-12T16:09:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:09:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:09:07.381 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-12T16:09:07.381 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:09:07.381 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-12T16:09:07.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-12T16:09:07.382 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-12T16:09:07.386 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-12T16:09:07.386 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-12T16:09:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 24425472 bytes 2024-09-12T16:09:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 40006656 bytes 2024-09-12T16:09:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 53680128 bytes 2024-09-12T16:09:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 99390464 bytes 2024-09-12T16:09:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 121036800 bytes 2024-09-12T16:09:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 135558144 bytes 2024-09-12T16:09:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 160167936 bytes 2024-09-12T16:09:08.372 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T16:09:08.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T16:09:08.372 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-12T16:09:08.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:09:08.372+0000 7fe568017640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 44403) UID: 1000 2024-09-12T16:09:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 199346176 bytes 2024-09-12T16:09:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 238610432 bytes 2024-09-12T16:09:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 282383360 bytes 2024-09-12T16:09:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 294883328 bytes 2024-09-12T16:09:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 333698048 bytes 2024-09-12T16:09:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 343794688 bytes 2024-09-12T16:09:09.373 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T16:09:09.373 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:09:09.374 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T16:09:09.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T16:09:09.374 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T16:09:09.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T16:09:09.377 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-09-12T16:09:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 423420928 bytes 2024-09-12T16:09:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 484391936 bytes 2024-09-12T16:09:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 578501632 bytes 2024-09-12T16:09:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 592386048 bytes 2024-09-12T16:09:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 621667328 bytes 2024-09-12T16:09:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 702116864 bytes 2024-09-12T16:09:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 773319680 bytes 2024-09-12T16:09:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 803717120 bytes 2024-09-12T16:09:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 881810432 bytes 2024-09-12T16:09:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 924551168 bytes 2024-09-12T16:09:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 946279424 bytes 2024-09-12T16:09:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 997647360 bytes 2024-09-12T16:09:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1023197184 bytes 2024-09-12T16:09:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1061897216 bytes 2024-09-12T16:09:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1068748800 bytes 2024-09-12T16:09:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1069366272 bytes 2024-09-12T16:09:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1099240448 bytes 2024-09-12T16:09:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1124629504 bytes 2024-09-12T16:09:12.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1154668544 bytes 2024-09-12T16:09:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1182555136 bytes 2024-09-12T16:09:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1222116352 bytes 2024-09-12T16:09:13.230 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1274120192 bytes 2024-09-12T16:09:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1295388672 bytes 2024-09-12T16:09:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1383742464 bytes 2024-09-12T16:09:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1409010688 bytes 2024-09-12T16:09:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1443311616 bytes 2024-09-12T16:09:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1487850496 bytes 2024-09-12T16:09:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1520999424 bytes 2024-09-12T16:09:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1540170752 bytes 2024-09-12T16:09:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1599301632 bytes 2024-09-12T16:09:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1631843328 bytes 2024-09-12T16:09:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1674602496 bytes 2024-09-12T16:09:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1703869440 bytes 2024-09-12T16:09:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1774629888 bytes 2024-09-12T16:09:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1813902336 bytes 2024-09-12T16:09:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1871218688 bytes 2024-09-12T16:09:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1889132544 bytes 2024-09-12T16:09:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 1966833664 bytes 2024-09-12T16:09:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2048370688 bytes 2024-09-12T16:09:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2110319616 bytes 2024-09-12T16:09:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2112013312 bytes 2024-09-12T16:09:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2141022208 bytes 2024-09-12T16:09:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2199785472 bytes 2024-09-12T16:09:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2266852352 bytes 2024-09-12T16:09:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2305888256 bytes 2024-09-12T16:09:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2401929216 bytes 2024-09-12T16:09:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2472440832 bytes 2024-09-12T16:09:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2492231680 bytes 2024-09-12T16:09:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2548330496 bytes 2024-09-12T16:09:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2617334784 bytes 2024-09-12T16:09:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2690931712 bytes 2024-09-12T16:09:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2769679360 bytes 2024-09-12T16:09:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2818588672 bytes 2024-09-12T16:09:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2857627648 bytes 2024-09-12T16:09:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2890596352 bytes 2024-09-12T16:09:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2947471360 bytes 2024-09-12T16:09:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2976614400 bytes 2024-09-12T16:09:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 3005115392 bytes 2024-09-12T16:09:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3019097088 bytes 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_17 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:09:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:09:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:09:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-09-12T16:09:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-12T16:09:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:09:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_17': 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13414786a33b 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13414786a33b 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:09:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:20 2024 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:20 2024 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:20 2024 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 3d399251-fa2f-40f2-9b35-dfd5e3c5d326 2024-09-12T16:09:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:09:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:09:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-12T16:09:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:09:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:09:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:09:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:09:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:09:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 27271168 bytes 2024-09-12T16:09:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 52033536 bytes 2024-09-12T16:09:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 105044992 bytes 2024-09-12T16:09:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 119771136 bytes 2024-09-12T16:09:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 139167744 bytes 2024-09-12T16:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 180500480 bytes 2024-09-12T16:09:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 237059072 bytes 2024-09-12T16:09:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 277667840 bytes 2024-09-12T16:09:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 324554752 bytes 2024-09-12T16:09:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 381943808 bytes 2024-09-12T16:09:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 456554496 bytes 2024-09-12T16:09:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 460536832 bytes 2024-09-12T16:09:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 481419264 bytes 2024-09-12T16:09:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 493412352 bytes 2024-09-12T16:09:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 552178688 bytes 2024-09-12T16:09:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 580142080 bytes 2024-09-12T16:09:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 580775936 bytes 2024-09-12T16:09:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 618844160 bytes 2024-09-12T16:09:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 672653312 bytes 2024-09-12T16:09:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 681741312 bytes 2024-09-12T16:09:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 722341888 bytes 2024-09-12T16:09:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 728597504 bytes 2024-09-12T16:09:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 769689600 bytes 2024-09-12T16:09:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 815013888 bytes 2024-09-12T16:09:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 866294784 bytes 2024-09-12T16:09:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 890773504 bytes 2024-09-12T16:09:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 891937792 bytes 2024-09-12T16:09:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 896449536 bytes 2024-09-12T16:09:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 897081344 bytes 2024-09-12T16:09:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 946580480 bytes 2024-09-12T16:09:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:09:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:09:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:09:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_18 2024-09-12T16:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-12T16:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-09-12T16:09:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:09:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:09:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-12T16:09:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:09:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-12T16:09:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:09:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_18': 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 134d294696bc 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.134d294696bc 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:09:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:26 2024 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:26 2024 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:26 2024 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 77378832-95d1-4af4-a433-6685e1d5dd58 2024-09-12T16:09:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:09:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:09:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-12T16:09:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:09:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:09:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:09:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:09:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:09:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 93486080 bytes 2024-09-12T16:09:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 149940224 bytes 2024-09-12T16:09:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 172883968 bytes 2024-09-12T16:09:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 210114560 bytes 2024-09-12T16:09:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 247001088 bytes 2024-09-12T16:09:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 261497856 bytes 2024-09-12T16:09:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 319636480 bytes 2024-09-12T16:09:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 408049664 bytes 2024-09-12T16:09:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 458589184 bytes 2024-09-12T16:09:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 512226304 bytes 2024-09-12T16:09:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 590476288 bytes 2024-09-12T16:09:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 642173952 bytes 2024-09-12T16:09:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 680341504 bytes 2024-09-12T16:09:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 743697408 bytes 2024-09-12T16:09:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 790730752 bytes 2024-09-12T16:09:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 833177600 bytes 2024-09-12T16:09:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 899523584 bytes 2024-09-12T16:09:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 938182656 bytes 2024-09-12T16:09:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 987056128 bytes 2024-09-12T16:09:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1014110208 bytes 2024-09-12T16:09:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1051100160 bytes 2024-09-12T16:09:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1080643584 bytes 2024-09-12T16:09:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1170615296 bytes 2024-09-12T16:09:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1222255616 bytes 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_19 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:09:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:09:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:09:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:09:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-12T16:09:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:09:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-12T16:09:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_19': 2024-09-12T16:09:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:09:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:09:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:09:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13592f85be7 2024-09-12T16:09:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13592f85be7 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:32 2024 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:32 2024 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:32 2024 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 2152ea11-154f-4401-9170-49378499a08c 2024-09-12T16:09:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:09:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:09:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-12T16:09:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:09:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:09:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=6 2024-09-12T16:09:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:09:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:09:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 69111808 bytes 2024-09-12T16:09:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 95792128 bytes 2024-09-12T16:09:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 164668416 bytes 2024-09-12T16:09:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 221836288 bytes 2024-09-12T16:09:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 298854400 bytes 2024-09-12T16:09:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 345353216 bytes 2024-09-12T16:09:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 428721152 bytes 2024-09-12T16:09:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 469988352 bytes 2024-09-12T16:09:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 553224192 bytes 2024-09-12T16:09:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 572643328 bytes 2024-09-12T16:09:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 621869056 bytes 2024-09-12T16:09:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 652607488 bytes 2024-09-12T16:09:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 742398976 bytes 2024-09-12T16:09:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 793701376 bytes 2024-09-12T16:09:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 834548736 bytes 2024-09-12T16:09:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 839201792 bytes 2024-09-12T16:09:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 910264320 bytes 2024-09-12T16:09:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 1003776000 bytes 2024-09-12T16:09:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 1058079744 bytes 2024-09-12T16:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1060655104 bytes 2024-09-12T16:09:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1095932928 bytes 2024-09-12T16:09:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1143739392 bytes 2024-09-12T16:09:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1183002624 bytes 2024-09-12T16:09:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1268659200 bytes 2024-09-12T16:09:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1315425280 bytes 2024-09-12T16:09:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1316038656 bytes 2024-09-12T16:09:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1339761664 bytes 2024-09-12T16:09:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1376897024 bytes 2024-09-12T16:09:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1423812608 bytes 2024-09-12T16:09:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1466397696 bytes 2024-09-12T16:09:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1526152192 bytes 2024-09-12T16:09:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:09:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_20 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-09-12T16:09:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:09:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:09:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-12T16:09:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:09:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-12T16:09:40.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:09:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-12T16:09:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_20': 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 136561da3e01 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.136561da3e01 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:39 2024 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:39 2024 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:39 2024 2024-09-12T16:09:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:09:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:09:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: ac9510df-01cf-4e79-97cb-6868ae398ef5 2024-09-12T16:09:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:09:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:09:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-12T16:09:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:09:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:09:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:09:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:09:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:09:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 27634688 bytes 2024-09-12T16:09:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 105940992 bytes 2024-09-12T16:09:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 134493184 bytes 2024-09-12T16:09:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 172045312 bytes 2024-09-12T16:09:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 193389568 bytes 2024-09-12T16:09:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 204420096 bytes 2024-09-12T16:09:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 274935808 bytes 2024-09-12T16:09:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 304300032 bytes 2024-09-12T16:09:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 360591360 bytes 2024-09-12T16:09:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 377041920 bytes 2024-09-12T16:09:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 488696832 bytes 2024-09-12T16:09:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 537874432 bytes 2024-09-12T16:09:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 588658688 bytes 2024-09-12T16:09:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 608781312 bytes 2024-09-12T16:09:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 617858048 bytes 2024-09-12T16:09:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 673456128 bytes 2024-09-12T16:09:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 722945024 bytes 2024-09-12T16:09:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 753057792 bytes 2024-09-12T16:09:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 837628928 bytes 2024-09-12T16:09:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 881456128 bytes 2024-09-12T16:09:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 915271680 bytes 2024-09-12T16:09:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1009487872 bytes 2024-09-12T16:09:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1033696256 bytes 2024-09-12T16:09:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1092004864 bytes 2024-09-12T16:09:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1132548096 bytes 2024-09-12T16:09:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1206395904 bytes 2024-09-12T16:09:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1263662080 bytes 2024-09-12T16:09:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1316476928 bytes 2024-09-12T16:09:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1330537472 bytes 2024-09-12T16:09:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1367709696 bytes 2024-09-12T16:09:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1377938432 bytes 2024-09-12T16:09:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1418366976 bytes 2024-09-12T16:09:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1419000832 bytes 2024-09-12T16:09:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1452498944 bytes 2024-09-12T16:09:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1466169344 bytes 2024-09-12T16:09:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1497648128 bytes 2024-09-12T16:09:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1498277888 bytes 2024-09-12T16:09:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1511144448 bytes 2024-09-12T16:09:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1590520832 bytes 2024-09-12T16:09:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1661819904 bytes 2024-09-12T16:09:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1692998656 bytes 2024-09-12T16:09:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1754886144 bytes 2024-09-12T16:09:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1807391744 bytes 2024-09-12T16:09:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1845037056 bytes 2024-09-12T16:09:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1882131456 bytes 2024-09-12T16:09:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1905192960 bytes 2024-09-12T16:09:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1990429696 bytes 2024-09-12T16:09:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2049421312 bytes 2024-09-12T16:09:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2081408000 bytes 2024-09-12T16:09:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2095773696 bytes 2024-09-12T16:09:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2139779072 bytes 2024-09-12T16:09:50.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2153709568 bytes 2024-09-12T16:09:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2154323968 bytes 2024-09-12T16:09:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2239656960 bytes 2024-09-12T16:09:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2282557440 bytes 2024-09-12T16:09:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2315428864 bytes 2024-09-12T16:09:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2362657792 bytes 2024-09-12T16:09:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2397766656 bytes 2024-09-12T16:09:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2409907200 bytes 2024-09-12T16:09:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_21 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:09:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:09:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:09:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-09-12T16:09:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:09:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:09:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:09:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-12T16:09:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:09:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:09:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-12T16:09:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:09:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_21': 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 137121cba75f 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.137121cba75f 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:09:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:51 2024 2024-09-12T16:09:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:51 2024 2024-09-12T16:09:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:51 2024 2024-09-12T16:09:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:09:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:09:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a8bf7886-b7c1-43a8-820b-159e65038b93 2024-09-12T16:09:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:09:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:09:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-12T16:09:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:09:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:09:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:09:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:09:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=14 2024-09-12T16:09:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:09:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:09:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 54317056 bytes 2024-09-12T16:09:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 121031680 bytes 2024-09-12T16:09:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 166191104 bytes 2024-09-12T16:09:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 219354112 bytes 2024-09-12T16:09:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 255097856 bytes 2024-09-12T16:09:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 280399872 bytes 2024-09-12T16:09:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 362267648 bytes 2024-09-12T16:09:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 405968896 bytes 2024-09-12T16:09:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 423462912 bytes 2024-09-12T16:09:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 508383232 bytes 2024-09-12T16:09:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 524325888 bytes 2024-09-12T16:09:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 588702720 bytes 2024-09-12T16:09:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 633522176 bytes 2024-09-12T16:09:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 690111488 bytes 2024-09-12T16:09:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 690757632 bytes 2024-09-12T16:09:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 751003648 bytes 2024-09-12T16:09:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 794450944 bytes 2024-09-12T16:09:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 837501952 bytes 2024-09-12T16:09:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 880881664 bytes 2024-09-12T16:09:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 942414848 bytes 2024-09-12T16:09:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 986702848 bytes 2024-09-12T16:09:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1007569920 bytes 2024-09-12T16:09:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1063696384 bytes 2024-09-12T16:09:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1113502720 bytes 2024-09-12T16:09:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1146146816 bytes 2024-09-12T16:09:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1180495872 bytes 2024-09-12T16:09:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1236044800 bytes 2024-09-12T16:09:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1302072320 bytes 2024-09-12T16:09:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1360184320 bytes 2024-09-12T16:09:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1386519552 bytes 2024-09-12T16:09:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1423528960 bytes 2024-09-12T16:09:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1474462720 bytes 2024-09-12T16:09:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1507650560 bytes 2024-09-12T16:09:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1508259840 bytes 2024-09-12T16:09:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1528867840 bytes 2024-09-12T16:09:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1555473408 bytes 2024-09-12T16:09:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1590160384 bytes 2024-09-12T16:09:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1663245312 bytes 2024-09-12T16:09:59.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1727470592 bytes 2024-09-12T16:09:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1814214656 bytes 2024-09-12T16:09:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1837193216 bytes 2024-09-12T16:10:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1895622656 bytes 2024-09-12T16:10:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1930360832 bytes 2024-09-12T16:10:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1975026688 bytes 2024-09-12T16:10:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2000230400 bytes 2024-09-12T16:10:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2015614976 bytes 2024-09-12T16:10:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2086411264 bytes 2024-09-12T16:10:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2177369088 bytes 2024-09-12T16:10:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2240441344 bytes 2024-09-12T16:10:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2264492032 bytes 2024-09-12T16:10:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2357088256 bytes 2024-09-12T16:10:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2401917952 bytes 2024-09-12T16:10:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2500572160 bytes 2024-09-12T16:10:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2526676992 bytes 2024-09-12T16:10:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2550489088 bytes 2024-09-12T16:10:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2595857408 bytes 2024-09-12T16:10:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2640607232 bytes 2024-09-12T16:10:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2697148416 bytes 2024-09-12T16:10:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2727973888 bytes 2024-09-12T16:10:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2770555904 bytes 2024-09-12T16:10:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2791389184 bytes 2024-09-12T16:10:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2811068416 bytes 2024-09-12T16:10:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2879046656 bytes 2024-09-12T16:10:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2897278976 bytes 2024-09-12T16:10:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2952291328 bytes 2024-09-12T16:10:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2989059072 bytes 2024-09-12T16:10:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 3001141248 bytes 2024-09-12T16:10:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 3025505280 bytes 2024-09-12T16:10:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 3031201792 bytes 2024-09-12T16:10:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 3107799040 bytes 2024-09-12T16:10:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 3183765504 bytes 2024-09-12T16:10:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3228395520 bytes 2024-09-12T16:10:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 3235426304 bytes 2024-09-12T16:10:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 3291412480 bytes 2024-09-12T16:10:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:10:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:10:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:10:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:10:06.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_22 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-12T16:10:06.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-09-12T16:10:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-12T16:10:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:10:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_22': 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 137dcebb22c3 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.137dcebb22c3 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:10:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:10:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:06 2024 2024-09-12T16:10:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:06 2024 2024-09-12T16:10:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:06 2024 2024-09-12T16:10:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:10:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:10:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 051e4edc-7365-4ada-b234-2ea4450f9d6d 2024-09-12T16:10:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:10:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:10:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-12T16:10:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:10:07.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:10:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=10 2024-09-12T16:10:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:10:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:10:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 81703936 bytes 2024-09-12T16:10:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 141470720 bytes 2024-09-12T16:10:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 238864384 bytes 2024-09-12T16:10:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 270971904 bytes 2024-09-12T16:10:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 332893184 bytes 2024-09-12T16:10:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 365924352 bytes 2024-09-12T16:10:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 378796032 bytes 2024-09-12T16:10:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 425259008 bytes 2024-09-12T16:10:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 441213952 bytes 2024-09-12T16:10:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 441827328 bytes 2024-09-12T16:10:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 465305600 bytes 2024-09-12T16:10:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 545871872 bytes 2024-09-12T16:10:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 560845824 bytes 2024-09-12T16:10:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 586215424 bytes 2024-09-12T16:10:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 664741888 bytes 2024-09-12T16:10:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 729987072 bytes 2024-09-12T16:10:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 746562560 bytes 2024-09-12T16:10:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 777284608 bytes 2024-09-12T16:10:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 807998464 bytes 2024-09-12T16:10:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 884179968 bytes 2024-09-12T16:10:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 967041024 bytes 2024-09-12T16:10:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1007572992 bytes 2024-09-12T16:10:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1027827712 bytes 2024-09-12T16:10:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1080421376 bytes 2024-09-12T16:10:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1081030656 bytes 2024-09-12T16:10:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1106455552 bytes 2024-09-12T16:10:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1148630016 bytes 2024-09-12T16:10:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1251937280 bytes 2024-09-12T16:10:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1272991744 bytes 2024-09-12T16:10:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1298944000 bytes 2024-09-12T16:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1335271424 bytes 2024-09-12T16:10:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1379581952 bytes 2024-09-12T16:10:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1411923968 bytes 2024-09-12T16:10:13.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1465148416 bytes 2024-09-12T16:10:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1527186432 bytes 2024-09-12T16:10:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1573074944 bytes 2024-09-12T16:10:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1659086848 bytes 2024-09-12T16:10:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1725058048 bytes 2024-09-12T16:10:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1775389696 bytes 2024-09-12T16:10:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1809122304 bytes 2024-09-12T16:10:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1829684224 bytes 2024-09-12T16:10:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1842882560 bytes 2024-09-12T16:10:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1942672384 bytes 2024-09-12T16:10:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2030374912 bytes 2024-09-12T16:10:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2079304704 bytes 2024-09-12T16:10:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2141948928 bytes 2024-09-12T16:10:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2160002048 bytes 2024-09-12T16:10:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2160589824 bytes 2024-09-12T16:10:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2222448640 bytes 2024-09-12T16:10:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2277156864 bytes 2024-09-12T16:10:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2323595264 bytes 2024-09-12T16:10:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2324213760 bytes 2024-09-12T16:10:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2461230080 bytes 2024-09-12T16:10:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:10:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_23 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:10:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:10:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:10:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:10:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:10:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-09-12T16:10:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:10:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:10:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-12T16:10:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:10:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-12T16:10:18.386 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9380381206713309) > weight (0.25) 2024-09-12T16:10:18.387 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7223677944169433) > weight (0.25) 2024-09-12T16:10:18.387 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6245814712476171) > weight (0.25) 2024-09-12T16:10:18.387 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8259178256325455) > weight (0.25) 2024-09-12T16:10:18.388 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-12T16:10:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:10:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-12T16:10:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_23': 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1389a5c4dce7 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1389a5c4dce7 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:10:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:17 2024 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:17 2024 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:17 2024 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: d6fec4d2-a64c-49a2-811d-33983c8062c0 2024-09-12T16:10:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:10:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:10:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-12T16:10:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:10:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:10:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T16:10:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:10:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:10:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 48053248 bytes 2024-09-12T16:10:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 66727936 bytes 2024-09-12T16:10:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 142575616 bytes 2024-09-12T16:10:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 168031232 bytes 2024-09-12T16:10:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 186780672 bytes 2024-09-12T16:10:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 249784320 bytes 2024-09-12T16:10:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 302842880 bytes 2024-09-12T16:10:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 352458752 bytes 2024-09-12T16:10:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 394541056 bytes 2024-09-12T16:10:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 402203648 bytes 2024-09-12T16:10:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 402811904 bytes 2024-09-12T16:10:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 420694016 bytes 2024-09-12T16:10:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 439766016 bytes 2024-09-12T16:10:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 490040320 bytes 2024-09-12T16:10:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 594395136 bytes 2024-09-12T16:10:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 624055296 bytes 2024-09-12T16:10:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 656747520 bytes 2024-09-12T16:10:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 724776960 bytes 2024-09-12T16:10:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 739645440 bytes 2024-09-12T16:10:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 781763584 bytes 2024-09-12T16:10:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 813216768 bytes 2024-09-12T16:10:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 834345984 bytes 2024-09-12T16:10:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 873020416 bytes 2024-09-12T16:10:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 889015296 bytes 2024-09-12T16:10:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 945458176 bytes 2024-09-12T16:10:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 960954368 bytes 2024-09-12T16:10:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1015055360 bytes 2024-09-12T16:10:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1057030144 bytes 2024-09-12T16:10:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1103506432 bytes 2024-09-12T16:10:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1160315904 bytes 2024-09-12T16:10:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1227768832 bytes 2024-09-12T16:10:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1268687872 bytes 2024-09-12T16:10:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1309899776 bytes 2024-09-12T16:10:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1376045056 bytes 2024-09-12T16:10:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1413109760 bytes 2024-09-12T16:10:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1438684160 bytes 2024-09-12T16:10:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1485887488 bytes 2024-09-12T16:10:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1580939264 bytes 2024-09-12T16:10:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1629289472 bytes 2024-09-12T16:10:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1651989504 bytes 2024-09-12T16:10:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1712289792 bytes 2024-09-12T16:10:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1784988672 bytes 2024-09-12T16:10:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1798578176 bytes 2024-09-12T16:10:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1839031296 bytes 2024-09-12T16:10:26.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1871049728 bytes 2024-09-12T16:10:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1921145856 bytes 2024-09-12T16:10:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1991291904 bytes 2024-09-12T16:10:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2053899264 bytes 2024-09-12T16:10:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2073547776 bytes 2024-09-12T16:10:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2114955264 bytes 2024-09-12T16:10:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2124037120 bytes 2024-09-12T16:10:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2134733824 bytes 2024-09-12T16:10:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2190866432 bytes 2024-09-12T16:10:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2233773056 bytes 2024-09-12T16:10:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2277473280 bytes 2024-09-12T16:10:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2342579200 bytes 2024-09-12T16:10:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2399617024 bytes 2024-09-12T16:10:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2412158976 bytes 2024-09-12T16:10:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2449746944 bytes 2024-09-12T16:10:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2466248704 bytes 2024-09-12T16:10:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2510559232 bytes 2024-09-12T16:10:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2553622528 bytes 2024-09-12T16:10:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2604564480 bytes 2024-09-12T16:10:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2661133312 bytes 2024-09-12T16:10:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2757228544 bytes 2024-09-12T16:10:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2858951680 bytes 2024-09-12T16:10:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2864060416 bytes 2024-09-12T16:10:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2884447232 bytes 2024-09-12T16:10:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:10:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:10:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:10:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:10:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_24 2024-09-12T16:10:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-09-12T16:10:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-12T16:10:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:10:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-12T16:10:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_24': 2024-09-12T16:10:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:10:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:10:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:10:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13953e210cd5 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13953e210cd5 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:30 2024 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:30 2024 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:30 2024 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 31fe714a-6461-4cbe-82ca-613754e1fba5 2024-09-12T16:10:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:10:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:10:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-12T16:10:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:10:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:10:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:10:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:10:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:10:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 56658944 bytes 2024-09-12T16:10:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 97607680 bytes 2024-09-12T16:10:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 130572288 bytes 2024-09-12T16:10:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 207242240 bytes 2024-09-12T16:10:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 221820928 bytes 2024-09-12T16:10:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 248108032 bytes 2024-09-12T16:10:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 280401920 bytes 2024-09-12T16:10:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 330892288 bytes 2024-09-12T16:10:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 378425344 bytes 2024-09-12T16:10:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 395883520 bytes 2024-09-12T16:10:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 403267584 bytes 2024-09-12T16:10:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 423990272 bytes 2024-09-12T16:10:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 438234112 bytes 2024-09-12T16:10:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 501307392 bytes 2024-09-12T16:10:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 552500224 bytes 2024-09-12T16:10:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 639364096 bytes 2024-09-12T16:10:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 671472640 bytes 2024-09-12T16:10:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 767201280 bytes 2024-09-12T16:10:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 851684352 bytes 2024-09-12T16:10:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 908582912 bytes 2024-09-12T16:10:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 966121472 bytes 2024-09-12T16:10:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 999733248 bytes 2024-09-12T16:10:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1020256256 bytes 2024-09-12T16:10:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1048052736 bytes 2024-09-12T16:10:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1106804736 bytes 2024-09-12T16:10:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1177897984 bytes 2024-09-12T16:10:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1210854400 bytes 2024-09-12T16:10:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1248007168 bytes 2024-09-12T16:10:36.377 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T16:10:36.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T16:10:36.378 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-12T16:10:36.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:10:36.378+0000 7ffa8cf8c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 48230) UID: 1000 2024-09-12T16:10:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1318614016 bytes 2024-09-12T16:10:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1368907776 bytes 2024-09-12T16:10:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1432858624 bytes 2024-09-12T16:10:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1485388800 bytes 2024-09-12T16:10:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1556883456 bytes 2024-09-12T16:10:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1557505024 bytes 2024-09-12T16:10:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1589975040 bytes 2024-09-12T16:10:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1654465536 bytes 2024-09-12T16:10:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1724204032 bytes 2024-09-12T16:10:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1792770048 bytes 2024-09-12T16:10:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1859259392 bytes 2024-09-12T16:10:38.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1911885824 bytes 2024-09-12T16:10:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1927203840 bytes 2024-09-12T16:10:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1984095232 bytes 2024-09-12T16:10:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2035877888 bytes 2024-09-12T16:10:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2082598912 bytes 2024-09-12T16:10:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2140907520 bytes 2024-09-12T16:10:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2176004096 bytes 2024-09-12T16:10:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2203311104 bytes 2024-09-12T16:10:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2244923392 bytes 2024-09-12T16:10:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2329466880 bytes 2024-09-12T16:10:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2398791680 bytes 2024-09-12T16:10:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2399411200 bytes 2024-09-12T16:10:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2428724224 bytes 2024-09-12T16:10:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2496291840 bytes 2024-09-12T16:10:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2523598848 bytes 2024-09-12T16:10:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2559507456 bytes 2024-09-12T16:10:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2611321856 bytes 2024-09-12T16:10:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2613192704 bytes 2024-09-12T16:10:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2715524096 bytes 2024-09-12T16:10:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2758543360 bytes 2024-09-12T16:10:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2799524864 bytes 2024-09-12T16:10:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2850374656 bytes 2024-09-12T16:10:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:10:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:10:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_25 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:10:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-09-12T16:10:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-12T16:10:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:10:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-12T16:10:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_25': 2024-09-12T16:10:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:10:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:10:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:10:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13a13c30164 2024-09-12T16:10:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13a13c30164 2024-09-12T16:10:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:42 2024 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:42 2024 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:42 2024 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: e4321129-6277-4549-af49-32f7bc842699 2024-09-12T16:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:10:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:10:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-12T16:10:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:10:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:10:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=14 2024-09-12T16:10:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:10:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:10:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 32212992 bytes 2024-09-12T16:10:43.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 128075776 bytes 2024-09-12T16:10:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 151264256 bytes 2024-09-12T16:10:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 168884224 bytes 2024-09-12T16:10:44.379 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T16:10:44.379 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:10:44.379 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T16:10:44.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T16:10:44.380 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T16:10:44.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T16:10:44.383 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-12T16:10:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 203925504 bytes 2024-09-12T16:10:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 233177088 bytes 2024-09-12T16:10:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 288570368 bytes 2024-09-12T16:10:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 360710144 bytes 2024-09-12T16:10:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 446451712 bytes 2024-09-12T16:10:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 517948416 bytes 2024-09-12T16:10:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 562101248 bytes 2024-09-12T16:10:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 589015040 bytes 2024-09-12T16:10:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 634721280 bytes 2024-09-12T16:10:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 687099904 bytes 2024-09-12T16:10:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 770310144 bytes 2024-09-12T16:10:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 829585408 bytes 2024-09-12T16:10:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 878404608 bytes 2024-09-12T16:10:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 883494912 bytes 2024-09-12T16:10:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 905643008 bytes 2024-09-12T16:10:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 915333120 bytes 2024-09-12T16:10:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 980769792 bytes 2024-09-12T16:10:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1000687616 bytes 2024-09-12T16:10:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1001316352 bytes 2024-09-12T16:10:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1039610880 bytes 2024-09-12T16:10:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1087370240 bytes 2024-09-12T16:10:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1191804928 bytes 2024-09-12T16:10:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1202599936 bytes 2024-09-12T16:10:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1241616384 bytes 2024-09-12T16:10:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1296053248 bytes 2024-09-12T16:10:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1335399424 bytes 2024-09-12T16:10:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1336011776 bytes 2024-09-12T16:10:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1392086016 bytes 2024-09-12T16:10:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1404564480 bytes 2024-09-12T16:10:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1505885184 bytes 2024-09-12T16:10:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1540672512 bytes 2024-09-12T16:10:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1608228864 bytes 2024-09-12T16:10:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1647729664 bytes 2024-09-12T16:10:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1688127488 bytes 2024-09-12T16:10:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1757102080 bytes 2024-09-12T16:10:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1773648896 bytes 2024-09-12T16:10:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1838929920 bytes 2024-09-12T16:10:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1929290752 bytes 2024-09-12T16:10:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1978268672 bytes 2024-09-12T16:10:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2025543680 bytes 2024-09-12T16:10:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2042870784 bytes 2024-09-12T16:10:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2118980608 bytes 2024-09-12T16:10:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2129819648 bytes 2024-09-12T16:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2155377664 bytes 2024-09-12T16:10:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2205582336 bytes 2024-09-12T16:10:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2240268288 bytes 2024-09-12T16:10:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2240885760 bytes 2024-09-12T16:10:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2284234752 bytes 2024-09-12T16:10:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2345010176 bytes 2024-09-12T16:10:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2396237824 bytes 2024-09-12T16:10:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2428502016 bytes 2024-09-12T16:10:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2501744640 bytes 2024-09-12T16:10:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2517575680 bytes 2024-09-12T16:10:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2579205120 bytes 2024-09-12T16:10:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2592624640 bytes 2024-09-12T16:10:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2652404736 bytes 2024-09-12T16:10:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2682461184 bytes 2024-09-12T16:10:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2771552256 bytes 2024-09-12T16:10:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2812710912 bytes 2024-09-12T16:10:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2844498944 bytes 2024-09-12T16:10:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2884214784 bytes 2024-09-12T16:10:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2933432320 bytes 2024-09-12T16:10:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2964379648 bytes 2024-09-12T16:10:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2967272448 bytes 2024-09-12T16:10:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 3008780288 bytes 2024-09-12T16:10:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 3111160832 bytes 2024-09-12T16:10:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 3157716992 bytes 2024-09-12T16:10:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3249582080 bytes 2024-09-12T16:10:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 3298187264 bytes 2024-09-12T16:10:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 3373125632 bytes 2024-09-12T16:10:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 3388263424 bytes 2024-09-12T16:10:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:10:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_26 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:10:57.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-09-12T16:10:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:10:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-12T16:10:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:10:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_26': 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13b14948a4dc 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13b14948a4dc 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:10:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:57 2024 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:57 2024 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:57 2024 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: de6e82d5-259a-465e-adb7-ef275c6429d6 2024-09-12T16:10:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:10:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:10:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-12T16:10:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:10:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:10:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:10:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:10:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=10 2024-09-12T16:10:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:10:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:10:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 40375296 bytes 2024-09-12T16:10:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 76214272 bytes 2024-09-12T16:10:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 109763584 bytes 2024-09-12T16:10:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 141149184 bytes 2024-09-12T16:10:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 181317632 bytes 2024-09-12T16:10:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 225087488 bytes 2024-09-12T16:10:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 235795456 bytes 2024-09-12T16:10:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 290437120 bytes 2024-09-12T16:10:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 337567744 bytes 2024-09-12T16:11:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 403374080 bytes 2024-09-12T16:11:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 427854848 bytes 2024-09-12T16:11:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 499686400 bytes 2024-09-12T16:11:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 522530816 bytes 2024-09-12T16:11:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 526642176 bytes 2024-09-12T16:11:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 568073216 bytes 2024-09-12T16:11:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 582257664 bytes 2024-09-12T16:11:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 604819456 bytes 2024-09-12T16:11:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 652920832 bytes 2024-09-12T16:11:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 697449472 bytes 2024-09-12T16:11:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 708024320 bytes 2024-09-12T16:11:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 720017408 bytes 2024-09-12T16:11:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 777054208 bytes 2024-09-12T16:11:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 817619968 bytes 2024-09-12T16:11:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 856822784 bytes 2024-09-12T16:11:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 863549440 bytes 2024-09-12T16:11:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 914425856 bytes 2024-09-12T16:11:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 989942784 bytes 2024-09-12T16:11:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1038152704 bytes 2024-09-12T16:11:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1085919232 bytes 2024-09-12T16:11:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1111356416 bytes 2024-09-12T16:11:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1124746240 bytes 2024-09-12T16:11:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1141757952 bytes 2024-09-12T16:11:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1245727744 bytes 2024-09-12T16:11:03.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1246345216 bytes 2024-09-12T16:11:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1262941184 bytes 2024-09-12T16:11:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1284945920 bytes 2024-09-12T16:11:04.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1326581760 bytes 2024-09-12T16:11:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1398838272 bytes 2024-09-12T16:11:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1446962176 bytes 2024-09-12T16:11:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1516014592 bytes 2024-09-12T16:11:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1589427200 bytes 2024-09-12T16:11:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1614969856 bytes 2024-09-12T16:11:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1616302080 bytes 2024-09-12T16:11:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1645441024 bytes 2024-09-12T16:11:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1726495744 bytes 2024-09-12T16:11:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1755558912 bytes 2024-09-12T16:11:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1794423808 bytes 2024-09-12T16:11:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1870061568 bytes 2024-09-12T16:11:06.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1946044416 bytes 2024-09-12T16:11:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1984176128 bytes 2024-09-12T16:11:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2030620672 bytes 2024-09-12T16:11:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2051020800 bytes 2024-09-12T16:11:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2112031744 bytes 2024-09-12T16:11:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2142278656 bytes 2024-09-12T16:11:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2192083968 bytes 2024-09-12T16:11:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2192703488 bytes 2024-09-12T16:11:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2206633984 bytes 2024-09-12T16:11:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2239539200 bytes 2024-09-12T16:11:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2243192832 bytes 2024-09-12T16:11:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:11:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:11:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:11:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:11:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_27 2024-09-12T16:11:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-12T16:11:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-09-12T16:11:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:11:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:11:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-12T16:11:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:11:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-12T16:11:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:11:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-12T16:11:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_27': 2024-09-12T16:11:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:11:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:11:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:11:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13ba5dc61b08 2024-09-12T16:11:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13ba5dc61b08 2024-09-12T16:11:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:11:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:11:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:11:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:11:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:08 2024 2024-09-12T16:11:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:08 2024 2024-09-12T16:11:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:08 2024 2024-09-12T16:11:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:11:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:11:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 2e3a6a3e-2072-4093-baab-81987b90abbb 2024-09-12T16:11:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:11:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:11:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-12T16:11:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:11:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:11:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=6 2024-09-12T16:11:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:11:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:11:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 19166208 bytes 2024-09-12T16:11:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 24058880 bytes 2024-09-12T16:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 86315008 bytes 2024-09-12T16:11:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 129140736 bytes 2024-09-12T16:11:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 173521920 bytes 2024-09-12T16:11:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 223158272 bytes 2024-09-12T16:11:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 235463680 bytes 2024-09-12T16:11:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 310293504 bytes 2024-09-12T16:11:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 345699328 bytes 2024-09-12T16:11:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 395979776 bytes 2024-09-12T16:11:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 444659712 bytes 2024-09-12T16:11:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 451152896 bytes 2024-09-12T16:11:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 509087744 bytes 2024-09-12T16:11:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 555213824 bytes 2024-09-12T16:11:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 576757760 bytes 2024-09-12T16:11:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 637391872 bytes 2024-09-12T16:11:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 676984832 bytes 2024-09-12T16:11:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 737339392 bytes 2024-09-12T16:11:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 770907136 bytes 2024-09-12T16:11:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 834769920 bytes 2024-09-12T16:11:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 876718080 bytes 2024-09-12T16:11:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 906013696 bytes 2024-09-12T16:11:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 985400320 bytes 2024-09-12T16:11:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1016614912 bytes 2024-09-12T16:11:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1064964096 bytes 2024-09-12T16:11:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1116642304 bytes 2024-09-12T16:11:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1159223296 bytes 2024-09-12T16:11:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1161206784 bytes 2024-09-12T16:11:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1173435392 bytes 2024-09-12T16:11:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1200564224 bytes 2024-09-12T16:11:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1233158144 bytes 2024-09-12T16:11:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1242338304 bytes 2024-09-12T16:11:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1272753152 bytes 2024-09-12T16:11:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1319803904 bytes 2024-09-12T16:11:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1337763840 bytes 2024-09-12T16:11:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_28 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:11:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:11:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:11:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:11:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-09-12T16:11:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:11:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:11:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-12T16:11:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:11:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-12T16:11:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:11:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-12T16:11:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_28': 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13c3dc8561b0 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13c3dc8561b0 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:11:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:15 2024 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:15 2024 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:15 2024 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 04c258ba-62b0-4f2a-9ccc-60bd5563c0a1 2024-09-12T16:11:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:11:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:11:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-12T16:11:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:11:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:11:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:11:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:11:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:11:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 20265984 bytes 2024-09-12T16:11:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 45252608 bytes 2024-09-12T16:11:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 77535232 bytes 2024-09-12T16:11:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 158050304 bytes 2024-09-12T16:11:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 215186432 bytes 2024-09-12T16:11:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 242813952 bytes 2024-09-12T16:11:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 265828352 bytes 2024-09-12T16:11:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 285644800 bytes 2024-09-12T16:11:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 360915968 bytes 2024-09-12T16:11:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 420079616 bytes 2024-09-12T16:11:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 435203072 bytes 2024-09-12T16:11:18.326 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:11:18.325+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5b8bb380 2024-09-12T16:11:18.326 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:11:18.325+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5bb8e8f0 2024-09-12T16:11:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 461251584 bytes 2024-09-12T16:11:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 500629504 bytes 2024-09-12T16:11:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 599749632 bytes 2024-09-12T16:11:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 619409408 bytes 2024-09-12T16:11:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 643202048 bytes 2024-09-12T16:11:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 701321216 bytes 2024-09-12T16:11:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 762919936 bytes 2024-09-12T16:11:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 763550720 bytes 2024-09-12T16:11:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 830799872 bytes 2024-09-12T16:11:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 894809088 bytes 2024-09-12T16:11:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 963216384 bytes 2024-09-12T16:11:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1003975680 bytes 2024-09-12T16:11:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1051061248 bytes 2024-09-12T16:11:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1052320768 bytes 2024-09-12T16:11:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1113153536 bytes 2024-09-12T16:11:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1154486272 bytes 2024-09-12T16:11:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1158122496 bytes 2024-09-12T16:11:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1232175104 bytes 2024-09-12T16:11:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1258288128 bytes 2024-09-12T16:11:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1326745600 bytes 2024-09-12T16:11:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1362472960 bytes 2024-09-12T16:11:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1418557440 bytes 2024-09-12T16:11:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1465185280 bytes 2024-09-12T16:11:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1503037440 bytes 2024-09-12T16:11:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1548846080 bytes 2024-09-12T16:11:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1596230656 bytes 2024-09-12T16:11:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1636398080 bytes 2024-09-12T16:11:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1733382144 bytes 2024-09-12T16:11:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1784509440 bytes 2024-09-12T16:11:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1833067520 bytes 2024-09-12T16:11:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1861843968 bytes 2024-09-12T16:11:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1984374784 bytes 2024-09-12T16:11:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2018125824 bytes 2024-09-12T16:11:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2062452736 bytes 2024-09-12T16:11:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2143693824 bytes 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_29 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:11:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:11:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:11:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:11:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:11:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:11:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-12T16:11:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:11:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_29': 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13cc636c05e8 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13cc636c05e8 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:11:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:11:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:25 2024 2024-09-12T16:11:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:25 2024 2024-09-12T16:11:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:25 2024 2024-09-12T16:11:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:11:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:11:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 9d01e9e8-f583-4610-9203-3eeb61dc6505 2024-09-12T16:11:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:11:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:11:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-12T16:11:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:11:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:11:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=13 2024-09-12T16:11:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:11:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:11:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 57101312 bytes 2024-09-12T16:11:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 103876608 bytes 2024-09-12T16:11:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 120818688 bytes 2024-09-12T16:11:27.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 160480256 bytes 2024-09-12T16:11:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 161093632 bytes 2024-09-12T16:11:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 179248128 bytes 2024-09-12T16:11:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 197753856 bytes 2024-09-12T16:11:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 246013952 bytes 2024-09-12T16:11:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 258280448 bytes 2024-09-12T16:11:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 294162432 bytes 2024-09-12T16:11:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 316509184 bytes 2024-09-12T16:11:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 375504896 bytes 2024-09-12T16:11:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 394834944 bytes 2024-09-12T16:11:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 419665920 bytes 2024-09-12T16:11:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 455981056 bytes 2024-09-12T16:11:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 515833856 bytes 2024-09-12T16:11:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 547623936 bytes 2024-09-12T16:11:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 582839296 bytes 2024-09-12T16:11:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 590685184 bytes 2024-09-12T16:11:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 640492544 bytes 2024-09-12T16:11:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 652047360 bytes 2024-09-12T16:11:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 732341248 bytes 2024-09-12T16:11:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 807477248 bytes 2024-09-12T16:11:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 856231936 bytes 2024-09-12T16:11:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 900328448 bytes 2024-09-12T16:11:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 962406400 bytes 2024-09-12T16:11:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1027131392 bytes 2024-09-12T16:11:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1094060032 bytes 2024-09-12T16:11:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1155011584 bytes 2024-09-12T16:11:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1203728384 bytes 2024-09-12T16:11:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1225895936 bytes 2024-09-12T16:11:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1307413504 bytes 2024-09-12T16:11:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1365072896 bytes 2024-09-12T16:11:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1430946816 bytes 2024-09-12T16:11:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1443440640 bytes 2024-09-12T16:11:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1479284736 bytes 2024-09-12T16:11:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1502843904 bytes 2024-09-12T16:11:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1529870336 bytes 2024-09-12T16:11:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1593943040 bytes 2024-09-12T16:11:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1632900096 bytes 2024-09-12T16:11:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1688593408 bytes 2024-09-12T16:11:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1750318080 bytes 2024-09-12T16:11:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1772818432 bytes 2024-09-12T16:11:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1841788928 bytes 2024-09-12T16:11:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1891061760 bytes 2024-09-12T16:11:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1919514624 bytes 2024-09-12T16:11:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1995589632 bytes 2024-09-12T16:11:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2026724352 bytes 2024-09-12T16:11:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2073722880 bytes 2024-09-12T16:11:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2119269376 bytes 2024-09-12T16:11:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2151878656 bytes 2024-09-12T16:11:35.388 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2781811674886897) > weight (0.25) 2024-09-12T16:11:35.388 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-12T16:11:35.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-12T16:11:35.389 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-12T16:11:35.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:11:35.388+0000 7fb8cdf56640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 48124) UID: 1000 2024-09-12T16:11:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2186953728 bytes 2024-09-12T16:11:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2277256192 bytes 2024-09-12T16:11:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2304727040 bytes 2024-09-12T16:11:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2339795968 bytes 2024-09-12T16:11:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2356390912 bytes 2024-09-12T16:11:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2418623488 bytes 2024-09-12T16:11:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2464423936 bytes 2024-09-12T16:11:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2531445760 bytes 2024-09-12T16:11:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2557085696 bytes 2024-09-12T16:11:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2638492672 bytes 2024-09-12T16:11:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2663936000 bytes 2024-09-12T16:11:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2715357184 bytes 2024-09-12T16:11:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2756301824 bytes 2024-09-12T16:11:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2771271680 bytes 2024-09-12T16:11:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2811196416 bytes 2024-09-12T16:11:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2815824896 bytes 2024-09-12T16:11:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2859615232 bytes 2024-09-12T16:11:38.390 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-12T16:11:38.390 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:11:38.391 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-12T16:11:38.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-12T16:11:38.391 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-12T16:11:38.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-12T16:11:38.394 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-12T16:11:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2916795392 bytes 2024-09-12T16:11:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2922824704 bytes 2024-09-12T16:11:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 2977170432 bytes 2024-09-12T16:11:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3007380480 bytes 2024-09-12T16:11:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 3068837888 bytes 2024-09-12T16:11:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 3101650944 bytes 2024-09-12T16:11:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:11:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:11:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:11:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:11:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_30 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-09-12T16:11:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:11:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:11:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-12T16:11:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:11:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-12T16:11:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:11:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_30': 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13df92656715 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13df92656715 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:11:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:39 2024 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:39 2024 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:39 2024 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 44d6b71b-d3f9-4302-8b48-4593d4e59e2c 2024-09-12T16:11:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:11:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:11:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-12T16:11:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:11:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:11:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:11:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:11:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:11:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 66526208 bytes 2024-09-12T16:11:41.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 132218880 bytes 2024-09-12T16:11:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 156847104 bytes 2024-09-12T16:11:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 190168064 bytes 2024-09-12T16:11:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 197435392 bytes 2024-09-12T16:11:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 219754496 bytes 2024-09-12T16:11:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 226900992 bytes 2024-09-12T16:11:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 301464576 bytes 2024-09-12T16:11:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 334898176 bytes 2024-09-12T16:11:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 362614784 bytes 2024-09-12T16:11:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 410113024 bytes 2024-09-12T16:11:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 471084032 bytes 2024-09-12T16:11:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 533723136 bytes 2024-09-12T16:11:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 553708544 bytes 2024-09-12T16:11:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 620702720 bytes 2024-09-12T16:11:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 703273984 bytes 2024-09-12T16:11:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 753776640 bytes 2024-09-12T16:11:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 773201920 bytes 2024-09-12T16:11:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 798795776 bytes 2024-09-12T16:11:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 868573184 bytes 2024-09-12T16:11:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 882803712 bytes 2024-09-12T16:11:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 926543872 bytes 2024-09-12T16:11:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 980328448 bytes 2024-09-12T16:11:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 994572288 bytes 2024-09-12T16:11:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1081066496 bytes 2024-09-12T16:11:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1112589312 bytes 2024-09-12T16:11:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1204151296 bytes 2024-09-12T16:11:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_31 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:11:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:11:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-12T16:11:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:11:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:11:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:11:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:11:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-09-12T16:11:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-12T16:11:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:11:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-12T16:11:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_31': 2024-09-12T16:11:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:11:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:11:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13e9499e204f 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13e9499e204f 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:45 2024 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:45 2024 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:45 2024 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 5666a566-1e64-44f3-803d-958a51c3829a 2024-09-12T16:11:46.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:11:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:11:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-12T16:11:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:11:46.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:11:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T16:11:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:11:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:11:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 54676480 bytes 2024-09-12T16:11:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 126676992 bytes 2024-09-12T16:11:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 162869248 bytes 2024-09-12T16:11:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 209051648 bytes 2024-09-12T16:11:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 252209152 bytes 2024-09-12T16:11:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 350360576 bytes 2024-09-12T16:11:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 392174592 bytes 2024-09-12T16:11:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 399835136 bytes 2024-09-12T16:11:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 438891520 bytes 2024-09-12T16:11:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 450468864 bytes 2024-09-12T16:11:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 551150592 bytes 2024-09-12T16:11:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 626479104 bytes 2024-09-12T16:11:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 695258112 bytes 2024-09-12T16:11:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 711668736 bytes 2024-09-12T16:11:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 751114240 bytes 2024-09-12T16:11:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 805118976 bytes 2024-09-12T16:11:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 827044864 bytes 2024-09-12T16:11:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 864764928 bytes 2024-09-12T16:11:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 888268800 bytes 2024-09-12T16:11:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 899562496 bytes 2024-09-12T16:11:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 913349632 bytes 2024-09-12T16:11:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 924871680 bytes 2024-09-12T16:11:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 972589056 bytes 2024-09-12T16:11:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 994214912 bytes 2024-09-12T16:11:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1092422656 bytes 2024-09-12T16:11:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1111906304 bytes 2024-09-12T16:11:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1215353856 bytes 2024-09-12T16:11:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1282293760 bytes 2024-09-12T16:11:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1316152320 bytes 2024-09-12T16:11:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1386188800 bytes 2024-09-12T16:11:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1431025664 bytes 2024-09-12T16:11:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1445250048 bytes 2024-09-12T16:11:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1454177280 bytes 2024-09-12T16:11:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1467061248 bytes 2024-09-12T16:11:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1508851712 bytes 2024-09-12T16:11:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1553686528 bytes 2024-09-12T16:11:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1610949632 bytes 2024-09-12T16:11:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1662840832 bytes 2024-09-12T16:11:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1751891968 bytes 2024-09-12T16:11:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1798270976 bytes 2024-09-12T16:11:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1821900800 bytes 2024-09-12T16:11:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1851311104 bytes 2024-09-12T16:11:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1868551168 bytes 2024-09-12T16:11:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1887019008 bytes 2024-09-12T16:11:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1944197120 bytes 2024-09-12T16:11:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2019986432 bytes 2024-09-12T16:11:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2046499840 bytes 2024-09-12T16:11:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2083964928 bytes 2024-09-12T16:11:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2138765312 bytes 2024-09-12T16:11:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2156834816 bytes 2024-09-12T16:11:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2224828416 bytes 2024-09-12T16:11:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2270940160 bytes 2024-09-12T16:11:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2353112064 bytes 2024-09-12T16:11:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2407368704 bytes 2024-09-12T16:11:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2421558272 bytes 2024-09-12T16:11:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2464091136 bytes 2024-09-12T16:11:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2490474496 bytes 2024-09-12T16:11:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2512695296 bytes 2024-09-12T16:11:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2546673664 bytes 2024-09-12T16:11:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2621385728 bytes 2024-09-12T16:11:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2660462592 bytes 2024-09-12T16:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2676881408 bytes 2024-09-12T16:11:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2705790976 bytes 2024-09-12T16:11:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2733934592 bytes 2024-09-12T16:11:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2811781120 bytes 2024-09-12T16:11:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:11:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:11:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:11:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:11:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_32 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:11:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:11:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-12T16:11:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:11:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-12T16:11:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_32': 2024-09-12T16:11:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:11:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:11:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:11:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13f2e7786fd4 2024-09-12T16:11:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13f2e7786fd4 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:58 2024 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:58 2024 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:58 2024 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:11:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 6d745883-fe82-4878-8b05-e9925084374f 2024-09-12T16:11:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:11:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:11:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-12T16:11:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:11:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:11:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:11:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:11:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=13 2024-09-12T16:11:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:11:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:11:59.383 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4840053813575611) > weight (0.25) 2024-09-12T16:11:59.384 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8852404741281101) > weight (0.25) 2024-09-12T16:11:59.384 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7519970015099601) > weight (0.25) 2024-09-12T16:11:59.384 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.44926802848650593) > weight (0.25) 2024-09-12T16:11:59.384 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2024-09-12T16:11:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 24100864 bytes 2024-09-12T16:11:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 103742464 bytes 2024-09-12T16:11:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 133278720 bytes 2024-09-12T16:12:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 188012544 bytes 2024-09-12T16:12:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 256184320 bytes 2024-09-12T16:12:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 294724608 bytes 2024-09-12T16:12:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 295337984 bytes 2024-09-12T16:12:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 328254464 bytes 2024-09-12T16:12:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 423086080 bytes 2024-09-12T16:12:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 515254272 bytes 2024-09-12T16:12:01.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 575263744 bytes 2024-09-12T16:12:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 617183232 bytes 2024-09-12T16:12:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 666230784 bytes 2024-09-12T16:12:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 696582144 bytes 2024-09-12T16:12:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 742203392 bytes 2024-09-12T16:12:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 801366016 bytes 2024-09-12T16:12:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 875126784 bytes 2024-09-12T16:12:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 949434368 bytes 2024-09-12T16:12:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 1046463488 bytes 2024-09-12T16:12:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1083671552 bytes 2024-09-12T16:12:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1172719616 bytes 2024-09-12T16:12:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1190061056 bytes 2024-09-12T16:12:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1245741056 bytes 2024-09-12T16:12:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1259573248 bytes 2024-09-12T16:12:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1293848576 bytes 2024-09-12T16:12:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1370470400 bytes 2024-09-12T16:12:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1386065920 bytes 2024-09-12T16:12:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1469077504 bytes 2024-09-12T16:12:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1512797184 bytes 2024-09-12T16:12:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1569227776 bytes 2024-09-12T16:12:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1617308672 bytes 2024-09-12T16:12:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1671026688 bytes 2024-09-12T16:12:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1722079232 bytes 2024-09-12T16:12:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1767943168 bytes 2024-09-12T16:12:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1788575744 bytes 2024-09-12T16:12:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1828298752 bytes 2024-09-12T16:12:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1828900864 bytes 2024-09-12T16:12:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1900475392 bytes 2024-09-12T16:12:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1928604672 bytes 2024-09-12T16:12:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1991906304 bytes 2024-09-12T16:12:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 2011196416 bytes 2024-09-12T16:12:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 2065359872 bytes 2024-09-12T16:12:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2110214144 bytes 2024-09-12T16:12:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2117215232 bytes 2024-09-12T16:12:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2156317696 bytes 2024-09-12T16:12:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2207653888 bytes 2024-09-12T16:12:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2290030592 bytes 2024-09-12T16:12:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2297242624 bytes 2024-09-12T16:12:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2343866368 bytes 2024-09-12T16:12:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2372217856 bytes 2024-09-12T16:12:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2437941248 bytes 2024-09-12T16:12:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2452237312 bytes 2024-09-12T16:12:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2491714560 bytes 2024-09-12T16:12:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2548213760 bytes 2024-09-12T16:12:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2631218176 bytes 2024-09-12T16:12:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2661745664 bytes 2024-09-12T16:12:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2703384576 bytes 2024-09-12T16:12:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2716581888 bytes 2024-09-12T16:12:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2758641664 bytes 2024-09-12T16:12:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2838334464 bytes 2024-09-12T16:12:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2847095808 bytes 2024-09-12T16:12:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2929217536 bytes 2024-09-12T16:12:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2969823232 bytes 2024-09-12T16:12:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 3018092544 bytes 2024-09-12T16:12:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 3104304128 bytes 2024-09-12T16:12:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 3133992960 bytes 2024-09-12T16:12:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 3191294976 bytes 2024-09-12T16:12:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 3263032320 bytes 2024-09-12T16:12:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_33 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:12:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-09-12T16:12:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:12:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:12:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-12T16:12:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:12:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-12T16:12:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:12:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-12T16:12:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_33': 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 13fa7c755239 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.13fa7c755239 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:12 2024 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:12 2024 2024-09-12T16:12:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:12 2024 2024-09-12T16:12:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:12:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:12:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 0817148d-e3ac-4c66-9784-73a94988f707 2024-09-12T16:12:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:12:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:12:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-12T16:12:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:12:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:12:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=7 2024-09-12T16:12:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:12:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:12:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 32889856 bytes 2024-09-12T16:12:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 73382912 bytes 2024-09-12T16:12:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 88079360 bytes 2024-09-12T16:12:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 149439488 bytes 2024-09-12T16:12:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 219535360 bytes 2024-09-12T16:12:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 277621760 bytes 2024-09-12T16:12:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 303572992 bytes 2024-09-12T16:12:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 407314432 bytes 2024-09-12T16:12:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 417748992 bytes 2024-09-12T16:12:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 451532800 bytes 2024-09-12T16:12:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 510141440 bytes 2024-09-12T16:12:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 588651520 bytes 2024-09-12T16:12:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 667547648 bytes 2024-09-12T16:12:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 679612416 bytes 2024-09-12T16:12:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 751764480 bytes 2024-09-12T16:12:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 776049664 bytes 2024-09-12T16:12:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 792039424 bytes 2024-09-12T16:12:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 859307008 bytes 2024-09-12T16:12:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 882243584 bytes 2024-09-12T16:12:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 903616512 bytes 2024-09-12T16:12:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1026573312 bytes 2024-09-12T16:12:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1124496384 bytes 2024-09-12T16:12:17.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1196963840 bytes 2024-09-12T16:12:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1197591552 bytes 2024-09-12T16:12:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1224828928 bytes 2024-09-12T16:12:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1285604352 bytes 2024-09-12T16:12:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1311645696 bytes 2024-09-12T16:12:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1324722176 bytes 2024-09-12T16:12:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1335977984 bytes 2024-09-12T16:12:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1346457600 bytes 2024-09-12T16:12:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1398042624 bytes 2024-09-12T16:12:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1428859904 bytes 2024-09-12T16:12:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1464404992 bytes 2024-09-12T16:12:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1482498048 bytes 2024-09-12T16:12:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1537933312 bytes 2024-09-12T16:12:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1566835712 bytes 2024-09-12T16:12:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1598111744 bytes 2024-09-12T16:12:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1614447616 bytes 2024-09-12T16:12:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:12:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:12:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:12:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:12:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_34 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:12:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:12:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:12:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:12:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-09-12T16:12:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:12:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:12:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-12T16:12:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:12:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-12T16:12:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:12:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_34': 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 140242e5880e 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.140242e5880e 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:12:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:12:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:20 2024 2024-09-12T16:12:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:20 2024 2024-09-12T16:12:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:20 2024 2024-09-12T16:12:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:12:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:12:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 1cffc7cb-c521-45a9-965f-d5265ebeee09 2024-09-12T16:12:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:12:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:12:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-12T16:12:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:12:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:12:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:12:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:12:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:12:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 1686528 bytes 2024-09-12T16:12:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 12018688 bytes 2024-09-12T16:12:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 93033472 bytes 2024-09-12T16:12:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 153875456 bytes 2024-09-12T16:12:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 227402752 bytes 2024-09-12T16:12:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 239160320 bytes 2024-09-12T16:12:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 275942400 bytes 2024-09-12T16:12:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 309792768 bytes 2024-09-12T16:12:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 314381312 bytes 2024-09-12T16:12:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 339615744 bytes 2024-09-12T16:12:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 392840192 bytes 2024-09-12T16:12:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 403744768 bytes 2024-09-12T16:12:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 420194304 bytes 2024-09-12T16:12:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 455918592 bytes 2024-09-12T16:12:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 484180992 bytes 2024-09-12T16:12:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 547040256 bytes 2024-09-12T16:12:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 583683072 bytes 2024-09-12T16:12:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 629906432 bytes 2024-09-12T16:12:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 728324096 bytes 2024-09-12T16:12:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 757949440 bytes 2024-09-12T16:12:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 883594240 bytes 2024-09-12T16:12:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 885482496 bytes 2024-09-12T16:12:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 916362240 bytes 2024-09-12T16:12:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 916973568 bytes 2024-09-12T16:12:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 982483968 bytes 2024-09-12T16:12:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1041438720 bytes 2024-09-12T16:12:26.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1058383872 bytes 2024-09-12T16:12:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_35 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:12:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:12:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:12:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-12T16:12:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:12:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_35': 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 140ac9ba69cc 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.140ac9ba69cc 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:12:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:26 2024 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:26 2024 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:26 2024 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 82a2b8c9-597b-4a1a-9ca6-24a0c920fc52 2024-09-12T16:12:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:12:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:12:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-12T16:12:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:12:27.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:12:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T16:12:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:12:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:12:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 607232 bytes 2024-09-12T16:12:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 37879808 bytes 2024-09-12T16:12:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 83407872 bytes 2024-09-12T16:12:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 133147648 bytes 2024-09-12T16:12:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 188728320 bytes 2024-09-12T16:12:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 236754944 bytes 2024-09-12T16:12:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 267301888 bytes 2024-09-12T16:12:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 291012608 bytes 2024-09-12T16:12:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 313657344 bytes 2024-09-12T16:12:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 336259072 bytes 2024-09-12T16:12:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 395207680 bytes 2024-09-12T16:12:29.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 448945152 bytes 2024-09-12T16:12:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 481270784 bytes 2024-09-12T16:12:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 557975552 bytes 2024-09-12T16:12:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 606169088 bytes 2024-09-12T16:12:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 675681280 bytes 2024-09-12T16:12:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 728503296 bytes 2024-09-12T16:12:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 776159232 bytes 2024-09-12T16:12:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 832809984 bytes 2024-09-12T16:12:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 887019520 bytes 2024-09-12T16:12:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 932533248 bytes 2024-09-12T16:12:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 958272512 bytes 2024-09-12T16:12:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1033026560 bytes 2024-09-12T16:12:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1048851456 bytes 2024-09-12T16:12:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1074088960 bytes 2024-09-12T16:12:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1112296448 bytes 2024-09-12T16:12:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1165656064 bytes 2024-09-12T16:12:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1187751936 bytes 2024-09-12T16:12:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1206038528 bytes 2024-09-12T16:12:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1227240448 bytes 2024-09-12T16:12:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1285019648 bytes 2024-09-12T16:12:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1328653312 bytes 2024-09-12T16:12:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1385324544 bytes 2024-09-12T16:12:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1442159616 bytes 2024-09-12T16:12:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1500950528 bytes 2024-09-12T16:12:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1578427392 bytes 2024-09-12T16:12:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1634788352 bytes 2024-09-12T16:12:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1671997440 bytes 2024-09-12T16:12:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1702582272 bytes 2024-09-12T16:12:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1727901696 bytes 2024-09-12T16:12:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1767913472 bytes 2024-09-12T16:12:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1844347904 bytes 2024-09-12T16:12:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1861940224 bytes 2024-09-12T16:12:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1903619072 bytes 2024-09-12T16:12:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1963963392 bytes 2024-09-12T16:12:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2004374528 bytes 2024-09-12T16:12:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2059785216 bytes 2024-09-12T16:12:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2084291584 bytes 2024-09-12T16:12:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2151081984 bytes 2024-09-12T16:12:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2240947200 bytes 2024-09-12T16:12:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2275087360 bytes 2024-09-12T16:12:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2305504256 bytes 2024-09-12T16:12:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2343363584 bytes 2024-09-12T16:12:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2364913664 bytes 2024-09-12T16:12:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2408819712 bytes 2024-09-12T16:12:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2453719040 bytes 2024-09-12T16:12:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2490826752 bytes 2024-09-12T16:12:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2557012992 bytes 2024-09-12T16:12:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2570138624 bytes 2024-09-12T16:12:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2606601216 bytes 2024-09-12T16:12:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2653194240 bytes 2024-09-12T16:12:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2717689856 bytes 2024-09-12T16:12:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2760758272 bytes 2024-09-12T16:12:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2805159936 bytes 2024-09-12T16:12:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:12:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_36 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-09-12T16:12:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:12:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:12:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-12T16:12:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:12:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-12T16:12:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:12:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-12T16:12:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_36': 2024-09-12T16:12:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:12:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:12:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:12:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 14121a7f2fd6 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.14121a7f2fd6 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:39 2024 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:39 2024 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:39 2024 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a0349dfc-0d36-4ede-9020-d1b706f0f9dc 2024-09-12T16:12:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:12:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:12:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-12T16:12:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:12:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:12:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:12:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:12:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:12:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 25157632 bytes 2024-09-12T16:12:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 76406784 bytes 2024-09-12T16:12:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 114131968 bytes 2024-09-12T16:12:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 133982208 bytes 2024-09-12T16:12:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 151013376 bytes 2024-09-12T16:12:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 178146304 bytes 2024-09-12T16:12:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 233226240 bytes 2024-09-12T16:12:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 274553856 bytes 2024-09-12T16:12:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 295904256 bytes 2024-09-12T16:12:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 358446080 bytes 2024-09-12T16:12:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 364162048 bytes 2024-09-12T16:12:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 389709824 bytes 2024-09-12T16:12:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 397287424 bytes 2024-09-12T16:12:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 420508672 bytes 2024-09-12T16:12:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 434998272 bytes 2024-09-12T16:12:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 517866496 bytes 2024-09-12T16:12:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 533272576 bytes 2024-09-12T16:12:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 601277440 bytes 2024-09-12T16:12:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 615742464 bytes 2024-09-12T16:12:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 656774144 bytes 2024-09-12T16:12:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 663839744 bytes 2024-09-12T16:12:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 746799104 bytes 2024-09-12T16:12:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 753628160 bytes 2024-09-12T16:12:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 848775168 bytes 2024-09-12T16:12:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 905701376 bytes 2024-09-12T16:12:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 961269760 bytes 2024-09-12T16:12:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 998481920 bytes 2024-09-12T16:12:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1005847552 bytes 2024-09-12T16:12:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1074797568 bytes 2024-09-12T16:12:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1101497344 bytes 2024-09-12T16:12:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1127160832 bytes 2024-09-12T16:12:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1150611456 bytes 2024-09-12T16:12:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1251115008 bytes 2024-09-12T16:12:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1292825600 bytes 2024-09-12T16:12:46.396 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8242715935440204) > weight (0.25) 2024-09-12T16:12:46.396 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-12T16:12:46.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-12T16:12:46.396 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-12T16:12:46.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:12:46.396+0000 7facc7958640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 49584) UID: 1000 2024-09-12T16:12:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1300899840 bytes 2024-09-12T16:12:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1303095296 bytes 2024-09-12T16:12:46.633 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:12:46.632+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5cb47380 2024-09-12T16:12:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1308866560 bytes 2024-09-12T16:12:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1375989760 bytes 2024-09-12T16:12:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1376613376 bytes 2024-09-12T16:12:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1414302720 bytes 2024-09-12T16:12:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1478717440 bytes 2024-09-12T16:12:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1499235328 bytes 2024-09-12T16:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1610877952 bytes 2024-09-12T16:12:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1619184640 bytes 2024-09-12T16:12:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1643518976 bytes 2024-09-12T16:12:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1654804480 bytes 2024-09-12T16:12:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1702307840 bytes 2024-09-12T16:12:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1727109120 bytes 2024-09-12T16:12:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1801283584 bytes 2024-09-12T16:12:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1863561216 bytes 2024-09-12T16:12:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 1951862784 bytes 2024-09-12T16:12:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 1974292480 bytes 2024-09-12T16:12:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:12:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:12:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_37 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-09-12T16:12:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-12T16:12:50.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:12:50.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_37': 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 141abe487afb 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.141abe487afb 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:12:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:12:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:12:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:12:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:49 2024 2024-09-12T16:12:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:49 2024 2024-09-12T16:12:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:49 2024 2024-09-12T16:12:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:12:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:12:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 0b18fcd6-90f2-4536-82ab-2679001c84e1 2024-09-12T16:12:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:12:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:12:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-12T16:12:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:12:50.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:12:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:12:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:12:50.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:12:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 60043264 bytes 2024-09-12T16:12:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 83540992 bytes 2024-09-12T16:12:50.902 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 127964160 bytes 2024-09-12T16:12:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 144220160 bytes 2024-09-12T16:12:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 206149632 bytes 2024-09-12T16:12:51.397 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-12T16:12:51.397 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:12:51.398 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-12T16:12:51.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-12T16:12:51.398 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-12T16:12:51.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-12T16:12:51.402 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-09-12T16:12:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 250019840 bytes 2024-09-12T16:12:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 326220800 bytes 2024-09-12T16:12:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 384795648 bytes 2024-09-12T16:12:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 403481600 bytes 2024-09-12T16:12:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 442488832 bytes 2024-09-12T16:12:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 498987008 bytes 2024-09-12T16:12:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 520314880 bytes 2024-09-12T16:12:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 587840512 bytes 2024-09-12T16:12:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 654560256 bytes 2024-09-12T16:12:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 655172608 bytes 2024-09-12T16:12:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 670050304 bytes 2024-09-12T16:12:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 721880064 bytes 2024-09-12T16:12:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 791220224 bytes 2024-09-12T16:12:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 810991616 bytes 2024-09-12T16:12:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 849654784 bytes 2024-09-12T16:12:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 860226560 bytes 2024-09-12T16:12:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 909858816 bytes 2024-09-12T16:12:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 968083456 bytes 2024-09-12T16:12:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1057403904 bytes 2024-09-12T16:12:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1122651136 bytes 2024-09-12T16:12:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1203161088 bytes 2024-09-12T16:12:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1237474304 bytes 2024-09-12T16:12:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1279608832 bytes 2024-09-12T16:12:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1328689152 bytes 2024-09-12T16:12:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1404443648 bytes 2024-09-12T16:12:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1406308352 bytes 2024-09-12T16:12:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1448193024 bytes 2024-09-12T16:12:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1525215232 bytes 2024-09-12T16:12:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1576852480 bytes 2024-09-12T16:12:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1615501312 bytes 2024-09-12T16:12:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1693712384 bytes 2024-09-12T16:12:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1736165376 bytes 2024-09-12T16:12:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1763734528 bytes 2024-09-12T16:12:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1852219392 bytes 2024-09-12T16:12:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1886821376 bytes 2024-09-12T16:12:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1957071872 bytes 2024-09-12T16:12:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 2053855232 bytes 2024-09-12T16:12:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2097864704 bytes 2024-09-12T16:12:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2099997696 bytes 2024-09-12T16:12:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2131515392 bytes 2024-09-12T16:12:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2173805568 bytes 2024-09-12T16:12:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:12:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:12:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_38 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:12:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:12:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:12:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-12T16:13:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:13:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_38': 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 142b3e093848 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.142b3e093848 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:13:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:59 2024 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:59 2024 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:59 2024 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: e0e6fdb8-ef78-418e-9cc0-4afd4bd69e5b 2024-09-12T16:13:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:13:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:13:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-12T16:13:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:13:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:13:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:13:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:13:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:13:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 55322624 bytes 2024-09-12T16:13:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 105752576 bytes 2024-09-12T16:13:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 106344448 bytes 2024-09-12T16:13:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 159245312 bytes 2024-09-12T16:13:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 186556416 bytes 2024-09-12T16:13:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 199420928 bytes 2024-09-12T16:13:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 222743552 bytes 2024-09-12T16:13:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 275358720 bytes 2024-09-12T16:13:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 275965952 bytes 2024-09-12T16:13:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 308703232 bytes 2024-09-12T16:13:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 328435712 bytes 2024-09-12T16:13:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 370652160 bytes 2024-09-12T16:13:02.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 415326208 bytes 2024-09-12T16:13:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 453625856 bytes 2024-09-12T16:13:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 481518592 bytes 2024-09-12T16:13:03.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 580154368 bytes 2024-09-12T16:13:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 649156608 bytes 2024-09-12T16:13:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 670708736 bytes 2024-09-12T16:13:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 745668608 bytes 2024-09-12T16:13:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 883259392 bytes 2024-09-12T16:13:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 903564288 bytes 2024-09-12T16:13:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 921774080 bytes 2024-09-12T16:13:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 927301632 bytes 2024-09-12T16:13:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 958524416 bytes 2024-09-12T16:13:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 997687296 bytes 2024-09-12T16:13:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1076776960 bytes 2024-09-12T16:13:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1099323392 bytes 2024-09-12T16:13:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1099932672 bytes 2024-09-12T16:13:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1166366720 bytes 2024-09-12T16:13:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1217399808 bytes 2024-09-12T16:13:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1240007680 bytes 2024-09-12T16:13:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1370489856 bytes 2024-09-12T16:13:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1418954752 bytes 2024-09-12T16:13:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1447169024 bytes 2024-09-12T16:13:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1486534656 bytes 2024-09-12T16:13:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1501431808 bytes 2024-09-12T16:13:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1559987200 bytes 2024-09-12T16:13:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1664763904 bytes 2024-09-12T16:13:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1708764160 bytes 2024-09-12T16:13:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1754766336 bytes 2024-09-12T16:13:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1807197184 bytes 2024-09-12T16:13:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1872002048 bytes 2024-09-12T16:13:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1913316352 bytes 2024-09-12T16:13:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1919188992 bytes 2024-09-12T16:13:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1963755520 bytes 2024-09-12T16:13:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2036741120 bytes 2024-09-12T16:13:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2091233280 bytes 2024-09-12T16:13:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2140165120 bytes 2024-09-12T16:13:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2250571776 bytes 2024-09-12T16:13:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2284731392 bytes 2024-09-12T16:13:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2303783936 bytes 2024-09-12T16:13:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2352766976 bytes 2024-09-12T16:13:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2424224768 bytes 2024-09-12T16:13:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2462314496 bytes 2024-09-12T16:13:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2510748672 bytes 2024-09-12T16:13:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2519533568 bytes 2024-09-12T16:13:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2598792192 bytes 2024-09-12T16:13:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2633631744 bytes 2024-09-12T16:13:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2655627264 bytes 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_39 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-12T16:13:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:13:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:13:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:13:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:13:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-09-12T16:13:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:13:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:13:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-12T16:13:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:13:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-12T16:13:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:13:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-12T16:13:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_39': 2024-09-12T16:13:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:13:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:13:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1433d461cb0 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1433d461cb0 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:11 2024 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:11 2024 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:11 2024 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: d0fe3ef0-64f0-4fc2-be1f-5b61839e3452 2024-09-12T16:13:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-12T16:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:13:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:13:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:13:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:13:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:13:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 31936512 bytes 2024-09-12T16:13:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 74093568 bytes 2024-09-12T16:13:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 129456128 bytes 2024-09-12T16:13:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 147371008 bytes 2024-09-12T16:13:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 217668608 bytes 2024-09-12T16:13:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 247768064 bytes 2024-09-12T16:13:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 257724416 bytes 2024-09-12T16:13:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 321371136 bytes 2024-09-12T16:13:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 355650560 bytes 2024-09-12T16:13:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 406862848 bytes 2024-09-12T16:13:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 450222080 bytes 2024-09-12T16:13:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 497534976 bytes 2024-09-12T16:13:14.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 547970048 bytes 2024-09-12T16:13:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 576503808 bytes 2024-09-12T16:13:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 641226752 bytes 2024-09-12T16:13:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 710105088 bytes 2024-09-12T16:13:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 747287552 bytes 2024-09-12T16:13:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 863551488 bytes 2024-09-12T16:13:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 867544064 bytes 2024-09-12T16:13:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 941349888 bytes 2024-09-12T16:13:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 985848832 bytes 2024-09-12T16:13:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 992644096 bytes 2024-09-12T16:13:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1025277952 bytes 2024-09-12T16:13:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1095996416 bytes 2024-09-12T16:13:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1154876416 bytes 2024-09-12T16:13:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1208266752 bytes 2024-09-12T16:13:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:13:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:13:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_40 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-09-12T16:13:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:13:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:13:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-12T16:13:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:13:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-12T16:13:18.384 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4181567867785815) > weight (0.25) 2024-09-12T16:13:18.385 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4970102430333576) > weight (0.25) 2024-09-12T16:13:18.385 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.44604922400856073) > weight (0.25) 2024-09-12T16:13:18.385 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9263636221789688) > weight (0.25) 2024-09-12T16:13:18.385 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-12T16:13:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:13:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-12T16:13:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_40': 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 143bcee9ee03 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.143bcee9ee03 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:17 2024 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:17 2024 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:17 2024 2024-09-12T16:13:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:13:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:13:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: b72eb11e-35a3-4497-848c-fa02d50ba2c9 2024-09-12T16:13:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:13:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:13:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-12T16:13:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:13:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:13:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=10 2024-09-12T16:13:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:13:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:13:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 25957376 bytes 2024-09-12T16:13:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 27940864 bytes 2024-09-12T16:13:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 108854272 bytes 2024-09-12T16:13:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 159277056 bytes 2024-09-12T16:13:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 159890432 bytes 2024-09-12T16:13:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 257917952 bytes 2024-09-12T16:13:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 357682176 bytes 2024-09-12T16:13:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 428879872 bytes 2024-09-12T16:13:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 521122816 bytes 2024-09-12T16:13:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 560602112 bytes 2024-09-12T16:13:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 617551872 bytes 2024-09-12T16:13:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 618158080 bytes 2024-09-12T16:13:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 637191168 bytes 2024-09-12T16:13:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 711090176 bytes 2024-09-12T16:13:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 756017152 bytes 2024-09-12T16:13:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 763585536 bytes 2024-09-12T16:13:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 773899264 bytes 2024-09-12T16:13:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 856708096 bytes 2024-09-12T16:13:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 948549632 bytes 2024-09-12T16:13:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1026335744 bytes 2024-09-12T16:13:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1087059968 bytes 2024-09-12T16:13:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1102049280 bytes 2024-09-12T16:13:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1158170624 bytes 2024-09-12T16:13:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1165386752 bytes 2024-09-12T16:13:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1229513728 bytes 2024-09-12T16:13:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1281666048 bytes 2024-09-12T16:13:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1291275264 bytes 2024-09-12T16:13:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1358945280 bytes 2024-09-12T16:13:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1427562496 bytes 2024-09-12T16:13:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1492049920 bytes 2024-09-12T16:13:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1554102272 bytes 2024-09-12T16:13:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1575572480 bytes 2024-09-12T16:13:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1633089536 bytes 2024-09-12T16:13:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1672102912 bytes 2024-09-12T16:13:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1755413504 bytes 2024-09-12T16:13:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1763405824 bytes 2024-09-12T16:13:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1808084992 bytes 2024-09-12T16:13:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1882211328 bytes 2024-09-12T16:13:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1955371008 bytes 2024-09-12T16:13:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1990346752 bytes 2024-09-12T16:13:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 2106047488 bytes 2024-09-12T16:13:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 2160716800 bytes 2024-09-12T16:13:27.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2184615936 bytes 2024-09-12T16:13:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2196413440 bytes 2024-09-12T16:13:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2250882048 bytes 2024-09-12T16:13:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2284449792 bytes 2024-09-12T16:13:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2327152640 bytes 2024-09-12T16:13:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2327747584 bytes 2024-09-12T16:13:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2350331904 bytes 2024-09-12T16:13:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2408679424 bytes 2024-09-12T16:13:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2464829440 bytes 2024-09-12T16:13:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:13:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:13:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:13:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:13:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_41 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-09-12T16:13:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:13:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:13:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-12T16:13:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:13:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-12T16:13:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:13:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_41': 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1443b9c80611 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1443b9c80611 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:13:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:28 2024 2024-09-12T16:13:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:28 2024 2024-09-12T16:13:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:28 2024 2024-09-12T16:13:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:13:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:13:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 208f0c1a-c137-477d-8165-0c0b6213b01f 2024-09-12T16:13:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:13:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:13:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-12T16:13:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:13:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:13:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T16:13:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:13:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:13:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 47860736 bytes 2024-09-12T16:13:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 62982144 bytes 2024-09-12T16:13:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 85049344 bytes 2024-09-12T16:13:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 136056832 bytes 2024-09-12T16:13:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 180938752 bytes 2024-09-12T16:13:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 230673408 bytes 2024-09-12T16:13:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 321668096 bytes 2024-09-12T16:13:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 352687104 bytes 2024-09-12T16:13:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 419380224 bytes 2024-09-12T16:13:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 451038208 bytes 2024-09-12T16:13:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 490019840 bytes 2024-09-12T16:13:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 492878848 bytes 2024-09-12T16:13:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 544295936 bytes 2024-09-12T16:13:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 561306624 bytes 2024-09-12T16:13:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 629307392 bytes 2024-09-12T16:13:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 652635136 bytes 2024-09-12T16:13:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 747210752 bytes 2024-09-12T16:13:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 812486656 bytes 2024-09-12T16:13:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 813104128 bytes 2024-09-12T16:13:32.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 897688576 bytes 2024-09-12T16:13:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 936436736 bytes 2024-09-12T16:13:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 975703040 bytes 2024-09-12T16:13:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1044366336 bytes 2024-09-12T16:13:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1068090368 bytes 2024-09-12T16:13:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1167898624 bytes 2024-09-12T16:13:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1209083904 bytes 2024-09-12T16:13:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1268163584 bytes 2024-09-12T16:13:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1294108672 bytes 2024-09-12T16:13:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1341995008 bytes 2024-09-12T16:13:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1419323392 bytes 2024-09-12T16:13:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1440840704 bytes 2024-09-12T16:13:35.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1527483392 bytes 2024-09-12T16:13:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1634380800 bytes 2024-09-12T16:13:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1685992448 bytes 2024-09-12T16:13:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1741196288 bytes 2024-09-12T16:13:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1809611776 bytes 2024-09-12T16:13:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1825964032 bytes 2024-09-12T16:13:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1831888896 bytes 2024-09-12T16:13:36.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1872201728 bytes 2024-09-12T16:13:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1917924352 bytes 2024-09-12T16:13:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1938654208 bytes 2024-09-12T16:13:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1977459712 bytes 2024-09-12T16:13:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2089224192 bytes 2024-09-12T16:13:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2137203712 bytes 2024-09-12T16:13:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2214384640 bytes 2024-09-12T16:13:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2236236800 bytes 2024-09-12T16:13:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2257613824 bytes 2024-09-12T16:13:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2297437184 bytes 2024-09-12T16:13:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2334641152 bytes 2024-09-12T16:13:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2375922688 bytes 2024-09-12T16:13:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2418630656 bytes 2024-09-12T16:13:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2443820032 bytes 2024-09-12T16:13:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2506177536 bytes 2024-09-12T16:13:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2539658240 bytes 2024-09-12T16:13:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2615385088 bytes 2024-09-12T16:13:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2628275200 bytes 2024-09-12T16:13:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2670586880 bytes 2024-09-12T16:13:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2705614848 bytes 2024-09-12T16:13:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2755263488 bytes 2024-09-12T16:13:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2814725120 bytes 2024-09-12T16:13:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2847017984 bytes 2024-09-12T16:13:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2859761664 bytes 2024-09-12T16:13:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2870537216 bytes 2024-09-12T16:13:41.266 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2928453632 bytes 2024-09-12T16:13:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:13:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:13:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:13:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:13:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_42 2024-09-12T16:13:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-09-12T16:13:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-12T16:13:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:13:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-12T16:13:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_42': 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 144d2f96128d 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.144d2f96128d 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:41 2024 2024-09-12T16:13:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:41 2024 2024-09-12T16:13:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:41 2024 2024-09-12T16:13:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:13:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:13:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 13c21475-07e1-46bd-818c-3e23de114deb 2024-09-12T16:13:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:13:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:13:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 68819968 bytes 2024-09-12T16:13:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 114851840 bytes 2024-09-12T16:13:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 142003200 bytes 2024-09-12T16:13:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 237879296 bytes 2024-09-12T16:13:43.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 248650752 bytes 2024-09-12T16:13:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 316478464 bytes 2024-09-12T16:13:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 338641920 bytes 2024-09-12T16:13:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 376219648 bytes 2024-09-12T16:13:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 446619648 bytes 2024-09-12T16:13:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 471371776 bytes 2024-09-12T16:13:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 596425728 bytes 2024-09-12T16:13:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 677363712 bytes 2024-09-12T16:13:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 767522816 bytes 2024-09-12T16:13:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 803574784 bytes 2024-09-12T16:13:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 849174528 bytes 2024-09-12T16:13:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 849799168 bytes 2024-09-12T16:13:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 899956736 bytes 2024-09-12T16:13:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 936388608 bytes 2024-09-12T16:13:46.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 939727872 bytes 2024-09-12T16:13:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 948738048 bytes 2024-09-12T16:13:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 976648192 bytes 2024-09-12T16:13:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1029709824 bytes 2024-09-12T16:13:46.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1113194496 bytes 2024-09-12T16:13:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1168635904 bytes 2024-09-12T16:13:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1234885632 bytes 2024-09-12T16:13:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1245492224 bytes 2024-09-12T16:13:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_43 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:13:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:13:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:13:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-09-12T16:13:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:13:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-12T16:13:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:13:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-12T16:13:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_43': 2024-09-12T16:13:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:13:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:13:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:13:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 14573136b5f7 2024-09-12T16:13:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.14573136b5f7 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:47 2024 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:47 2024 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:47 2024 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 5e72a01d-2edd-4c16-ac2b-2b4399cf2dd2 2024-09-12T16:13:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-12T16:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:13:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:13:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=12 2024-09-12T16:13:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:13:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:13:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 18151424 bytes 2024-09-12T16:13:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 71239680 bytes 2024-09-12T16:13:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 135386112 bytes 2024-09-12T16:13:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 178160640 bytes 2024-09-12T16:13:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 245398528 bytes 2024-09-12T16:13:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 303336448 bytes 2024-09-12T16:13:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 331554816 bytes 2024-09-12T16:13:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 378858496 bytes 2024-09-12T16:13:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 388653056 bytes 2024-09-12T16:13:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 413638656 bytes 2024-09-12T16:13:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 459091968 bytes 2024-09-12T16:13:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 532935680 bytes 2024-09-12T16:13:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 571181056 bytes 2024-09-12T16:13:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 614056960 bytes 2024-09-12T16:13:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 666595328 bytes 2024-09-12T16:13:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 713054208 bytes 2024-09-12T16:13:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 747601920 bytes 2024-09-12T16:13:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 770727936 bytes 2024-09-12T16:13:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 785070080 bytes 2024-09-12T16:13:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 797075456 bytes 2024-09-12T16:13:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 826210304 bytes 2024-09-12T16:13:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 847300608 bytes 2024-09-12T16:13:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 902439936 bytes 2024-09-12T16:13:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 912644096 bytes 2024-09-12T16:13:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 936885248 bytes 2024-09-12T16:13:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1036156928 bytes 2024-09-12T16:13:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1072879616 bytes 2024-09-12T16:13:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1115881472 bytes 2024-09-12T16:13:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1160019968 bytes 2024-09-12T16:13:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1233622016 bytes 2024-09-12T16:13:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1299938304 bytes 2024-09-12T16:13:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1355419648 bytes 2024-09-12T16:13:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1404167168 bytes 2024-09-12T16:13:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1469685760 bytes 2024-09-12T16:13:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1473183744 bytes 2024-09-12T16:13:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1553771520 bytes 2024-09-12T16:13:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1582140416 bytes 2024-09-12T16:13:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1643498496 bytes 2024-09-12T16:13:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1690304512 bytes 2024-09-12T16:13:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1738332160 bytes 2024-09-12T16:13:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1748701184 bytes 2024-09-12T16:13:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1763227648 bytes 2024-09-12T16:13:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1820993536 bytes 2024-09-12T16:13:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1889169408 bytes 2024-09-12T16:13:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1954112512 bytes 2024-09-12T16:13:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2011365376 bytes 2024-09-12T16:13:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2045340672 bytes 2024-09-12T16:13:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2103709696 bytes 2024-09-12T16:13:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2108361728 bytes 2024-09-12T16:13:57.578 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:13:57.577+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5a806820 2024-09-12T16:13:57.578 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:13:57.577+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5b8bb380 2024-09-12T16:13:57.578 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:13:57.577+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5bb8e8f0 2024-09-12T16:13:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2254964736 bytes 2024-09-12T16:13:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2317056000 bytes 2024-09-12T16:13:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2368627712 bytes 2024-09-12T16:13:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2424352768 bytes 2024-09-12T16:13:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2480098304 bytes 2024-09-12T16:13:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2544430080 bytes 2024-09-12T16:13:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2573713408 bytes 2024-09-12T16:13:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2575272960 bytes 2024-09-12T16:13:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2590564352 bytes 2024-09-12T16:13:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2617456640 bytes 2024-09-12T16:13:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2667872256 bytes 2024-09-12T16:13:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2715406336 bytes 2024-09-12T16:13:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2737570816 bytes 2024-09-12T16:14:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2776053760 bytes 2024-09-12T16:14:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2795098112 bytes 2024-09-12T16:14:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2838785024 bytes 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_44 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:14:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:14:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:14:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-09-12T16:14:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:14:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:14:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-12T16:14:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:14:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-12T16:14:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:14:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-12T16:14:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_44': 2024-09-12T16:14:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:14:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:14:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:14:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1460435986b 2024-09-12T16:14:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1460435986b 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:00 2024 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:00 2024 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:00 2024 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 744e74ce-448f-4d0c-b2d1-b1a3be444e01 2024-09-12T16:14:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:14:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:14:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-12T16:14:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:14:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:14:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=8 2024-09-12T16:14:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:14:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:14:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 73908224 bytes 2024-09-12T16:14:02.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 94711808 bytes 2024-09-12T16:14:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 111842304 bytes 2024-09-12T16:14:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 125119488 bytes 2024-09-12T16:14:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 176687104 bytes 2024-09-12T16:14:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 245074944 bytes 2024-09-12T16:14:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 298779648 bytes 2024-09-12T16:14:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 371218432 bytes 2024-09-12T16:14:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 428876800 bytes 2024-09-12T16:14:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 470063104 bytes 2024-09-12T16:14:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 570117120 bytes 2024-09-12T16:14:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 593050624 bytes 2024-09-12T16:14:04.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 593665024 bytes 2024-09-12T16:14:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 611024896 bytes 2024-09-12T16:14:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 645224448 bytes 2024-09-12T16:14:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 688813056 bytes 2024-09-12T16:14:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 732136448 bytes 2024-09-12T16:14:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 767387648 bytes 2024-09-12T16:14:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 861562880 bytes 2024-09-12T16:14:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 885966848 bytes 2024-09-12T16:14:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 931960832 bytes 2024-09-12T16:14:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 932564992 bytes 2024-09-12T16:14:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 981657600 bytes 2024-09-12T16:14:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1015632896 bytes 2024-09-12T16:14:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1036450816 bytes 2024-09-12T16:14:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1037887488 bytes 2024-09-12T16:14:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1120193536 bytes 2024-09-12T16:14:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1175526400 bytes 2024-09-12T16:14:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1197586432 bytes 2024-09-12T16:14:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1258210304 bytes 2024-09-12T16:14:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1300350976 bytes 2024-09-12T16:14:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1356595200 bytes 2024-09-12T16:14:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1413120000 bytes 2024-09-12T16:14:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1452832768 bytes 2024-09-12T16:14:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1494209536 bytes 2024-09-12T16:14:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1531323392 bytes 2024-09-12T16:14:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1550627840 bytes 2024-09-12T16:14:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1624064000 bytes 2024-09-12T16:14:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1697521664 bytes 2024-09-12T16:14:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1730620416 bytes 2024-09-12T16:14:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1789245440 bytes 2024-09-12T16:14:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1840102400 bytes 2024-09-12T16:14:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1864251392 bytes 2024-09-12T16:14:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_45 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:14:09.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-09-12T16:14:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:14:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-12T16:14:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:14:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-12T16:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_45': 2024-09-12T16:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 146a4e6b3f76 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.146a4e6b3f76 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:09 2024 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:09 2024 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:09 2024 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: ad26f91c-a3fe-4c1d-910b-f56df2175de9 2024-09-12T16:14:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:14:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:14:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-12T16:14:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:14:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:14:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=10 2024-09-12T16:14:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:14:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:14:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 74318848 bytes 2024-09-12T16:14:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 149561344 bytes 2024-09-12T16:14:11.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 181504000 bytes 2024-09-12T16:14:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 218145792 bytes 2024-09-12T16:14:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 257941504 bytes 2024-09-12T16:14:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 290379776 bytes 2024-09-12T16:14:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 361358336 bytes 2024-09-12T16:14:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 399901696 bytes 2024-09-12T16:14:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 444248064 bytes 2024-09-12T16:14:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 494288896 bytes 2024-09-12T16:14:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 559971328 bytes 2024-09-12T16:14:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 613166080 bytes 2024-09-12T16:14:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 635323392 bytes 2024-09-12T16:14:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 696578048 bytes 2024-09-12T16:14:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 699085824 bytes 2024-09-12T16:14:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 729194496 bytes 2024-09-12T16:14:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 752645120 bytes 2024-09-12T16:14:13.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 805420032 bytes 2024-09-12T16:14:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 831063040 bytes 2024-09-12T16:14:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 833286144 bytes 2024-09-12T16:14:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 878763008 bytes 2024-09-12T16:14:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 928163840 bytes 2024-09-12T16:14:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1007343616 bytes 2024-09-12T16:14:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1037105152 bytes 2024-09-12T16:14:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1059004416 bytes 2024-09-12T16:14:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1152482304 bytes 2024-09-12T16:14:15.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1174294528 bytes 2024-09-12T16:14:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1259906048 bytes 2024-09-12T16:14:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1277488128 bytes 2024-09-12T16:14:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1367513088 bytes 2024-09-12T16:14:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1389900800 bytes 2024-09-12T16:14:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1425518592 bytes 2024-09-12T16:14:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1430516736 bytes 2024-09-12T16:14:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1456840704 bytes 2024-09-12T16:14:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1521583104 bytes 2024-09-12T16:14:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1577006080 bytes 2024-09-12T16:14:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1593254912 bytes 2024-09-12T16:14:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1611181056 bytes 2024-09-12T16:14:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1652034560 bytes 2024-09-12T16:14:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1713317888 bytes 2024-09-12T16:14:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1771795456 bytes 2024-09-12T16:14:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1816965120 bytes 2024-09-12T16:14:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1881345024 bytes 2024-09-12T16:14:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1910721536 bytes 2024-09-12T16:14:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1956814848 bytes 2024-09-12T16:14:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2022821888 bytes 2024-09-12T16:14:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2050756608 bytes 2024-09-12T16:14:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2082591744 bytes 2024-09-12T16:14:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2136057856 bytes 2024-09-12T16:14:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2153479168 bytes 2024-09-12T16:14:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2155953152 bytes 2024-09-12T16:14:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2267760640 bytes 2024-09-12T16:14:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2333510656 bytes 2024-09-12T16:14:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_46 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-09-12T16:14:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:14:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:14:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-12T16:14:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:14:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-12T16:14:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:14:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_46': 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 147332884382 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.147332884382 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:14:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:20 2024 2024-09-12T16:14:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:20 2024 2024-09-12T16:14:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:20 2024 2024-09-12T16:14:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:14:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:14:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c85a9f05-e741-40a3-80f4-86c92eb59802 2024-09-12T16:14:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:14:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:14:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-12T16:14:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:14:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:14:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:14:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:14:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:14:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 24611840 bytes 2024-09-12T16:14:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 73297920 bytes 2024-09-12T16:14:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 101325824 bytes 2024-09-12T16:14:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 185974784 bytes 2024-09-12T16:14:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 277049344 bytes 2024-09-12T16:14:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 289942528 bytes 2024-09-12T16:14:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 361526272 bytes 2024-09-12T16:14:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 382711808 bytes 2024-09-12T16:14:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 385538048 bytes 2024-09-12T16:14:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 444649472 bytes 2024-09-12T16:14:23.403 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5881343142837929) > weight (0.25) 2024-09-12T16:14:23.403 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6760005277871257) > weight (0.25) 2024-09-12T16:14:23.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2589061481910556) > weight (0.25) 2024-09-12T16:14:23.404 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.29387204925788935) > weight (0.25) 2024-09-12T16:14:23.404 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-12T16:14:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 502649856 bytes 2024-09-12T16:14:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 528229376 bytes 2024-09-12T16:14:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 566095872 bytes 2024-09-12T16:14:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 661132288 bytes 2024-09-12T16:14:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 701089792 bytes 2024-09-12T16:14:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 701696000 bytes 2024-09-12T16:14:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 743572480 bytes 2024-09-12T16:14:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 744197120 bytes 2024-09-12T16:14:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 769243136 bytes 2024-09-12T16:14:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 859530240 bytes 2024-09-12T16:14:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 886496256 bytes 2024-09-12T16:14:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 912905216 bytes 2024-09-12T16:14:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 950589440 bytes 2024-09-12T16:14:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1028282368 bytes 2024-09-12T16:14:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1101505536 bytes 2024-09-12T16:14:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1182654464 bytes 2024-09-12T16:14:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_47 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-09-12T16:14:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:14:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:14:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-12T16:14:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:14:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-12T16:14:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:14:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_47': 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 147bf81ca7ee 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.147bf81ca7ee 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:26 2024 2024-09-12T16:14:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:26 2024 2024-09-12T16:14:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:26 2024 2024-09-12T16:14:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:14:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:14:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: eac71c3a-77dc-48b2-a8e3-c7000c1abbd9 2024-09-12T16:14:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:14:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:14:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-12T16:14:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:14:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:14:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=6 2024-09-12T16:14:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:14:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:14:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 28423168 bytes 2024-09-12T16:14:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 43711488 bytes 2024-09-12T16:14:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 80847872 bytes 2024-09-12T16:14:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 134856704 bytes 2024-09-12T16:14:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 226291712 bytes 2024-09-12T16:14:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 312219648 bytes 2024-09-12T16:14:28.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 376289280 bytes 2024-09-12T16:14:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 385316864 bytes 2024-09-12T16:14:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 444622848 bytes 2024-09-12T16:14:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 474814464 bytes 2024-09-12T16:14:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 502939648 bytes 2024-09-12T16:14:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 559331328 bytes 2024-09-12T16:14:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 629252096 bytes 2024-09-12T16:14:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 678510592 bytes 2024-09-12T16:14:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 717688832 bytes 2024-09-12T16:14:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 748164096 bytes 2024-09-12T16:14:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 775971840 bytes 2024-09-12T16:14:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 847102976 bytes 2024-09-12T16:14:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 882208768 bytes 2024-09-12T16:14:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 919230464 bytes 2024-09-12T16:14:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 973280256 bytes 2024-09-12T16:14:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 974383104 bytes 2024-09-12T16:14:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1022431232 bytes 2024-09-12T16:14:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1085256704 bytes 2024-09-12T16:14:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1155930112 bytes 2024-09-12T16:14:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1168477184 bytes 2024-09-12T16:14:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1222884352 bytes 2024-09-12T16:14:32.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1288082432 bytes 2024-09-12T16:14:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1308484608 bytes 2024-09-12T16:14:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1364134912 bytes 2024-09-12T16:14:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1377107968 bytes 2024-09-12T16:14:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1411154944 bytes 2024-09-12T16:14:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_48 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:14:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:14:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:14:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-09-12T16:14:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:14:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:14:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-12T16:14:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:14:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-12T16:14:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:14:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-12T16:14:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_48': 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 148452ce38c0 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.148452ce38c0 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:33 2024 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:33 2024 2024-09-12T16:14:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:33 2024 2024-09-12T16:14:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:14:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:14:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 9815c7d5-2e9a-44c3-b4bc-6316340f81e4 2024-09-12T16:14:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:14:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:14:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-12T16:14:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:14:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:14:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:14:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:14:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:14:34.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 70400000 bytes 2024-09-12T16:14:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 109985792 bytes 2024-09-12T16:14:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 187020288 bytes 2024-09-12T16:14:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 253404160 bytes 2024-09-12T16:14:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 254004224 bytes 2024-09-12T16:14:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 307949568 bytes 2024-09-12T16:14:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 388123648 bytes 2024-09-12T16:14:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 425966592 bytes 2024-09-12T16:14:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 430486528 bytes 2024-09-12T16:14:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 479890432 bytes 2024-09-12T16:14:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 507789312 bytes 2024-09-12T16:14:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 535712768 bytes 2024-09-12T16:14:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 548812800 bytes 2024-09-12T16:14:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 622216192 bytes 2024-09-12T16:14:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 662697984 bytes 2024-09-12T16:14:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 722374656 bytes 2024-09-12T16:14:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 757582848 bytes 2024-09-12T16:14:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 841843712 bytes 2024-09-12T16:14:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 855613440 bytes 2024-09-12T16:14:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 877043712 bytes 2024-09-12T16:14:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 906624000 bytes 2024-09-12T16:14:38.386 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8340317578826122) > weight (0.25) 2024-09-12T16:14:38.386 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-12T16:14:38.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-12T16:14:38.387 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-12T16:14:38.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:14:38.387+0000 7f7b6cabf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 38081) UID: 1000 2024-09-12T16:14:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 962131968 bytes 2024-09-12T16:14:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1052096512 bytes 2024-09-12T16:14:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1109982208 bytes 2024-09-12T16:14:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1146278912 bytes 2024-09-12T16:14:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1218423808 bytes 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_49 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-12T16:14:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:14:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:14:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:14:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:14:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-09-12T16:14:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:14:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:14:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-12T16:14:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:14:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-12T16:14:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:14:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-12T16:14:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_49': 2024-09-12T16:14:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 148eb4d662f0 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.148eb4d662f0 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:39 2024 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:39 2024 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:39 2024 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:14:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:14:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: cb8151fa-d57f-4d3c-8f0f-b2e94e60e773 2024-09-12T16:14:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:14:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:14:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-12T16:14:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:14:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:14:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:14:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:14:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:14:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 39190528 bytes 2024-09-12T16:14:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 72517632 bytes 2024-09-12T16:14:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 85004288 bytes 2024-09-12T16:14:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 118686720 bytes 2024-09-12T16:14:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 139820032 bytes 2024-09-12T16:14:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 185246720 bytes 2024-09-12T16:14:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 253684736 bytes 2024-09-12T16:14:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 286290944 bytes 2024-09-12T16:14:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 345481216 bytes 2024-09-12T16:14:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 377224192 bytes 2024-09-12T16:14:42.387 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-12T16:14:42.388 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:14:42.388 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-12T16:14:42.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-12T16:14:42.388 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-12T16:14:42.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-12T16:14:42.391 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-12T16:14:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 445031424 bytes 2024-09-12T16:14:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 474782720 bytes 2024-09-12T16:14:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 522823680 bytes 2024-09-12T16:14:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 536537088 bytes 2024-09-12T16:14:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 568800256 bytes 2024-09-12T16:14:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 589576192 bytes 2024-09-12T16:14:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 607703040 bytes 2024-09-12T16:14:43.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 621684736 bytes 2024-09-12T16:14:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 659475456 bytes 2024-09-12T16:14:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 673161216 bytes 2024-09-12T16:14:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 723753984 bytes 2024-09-12T16:14:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 762509312 bytes 2024-09-12T16:14:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 777051136 bytes 2024-09-12T16:14:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 824321024 bytes 2024-09-12T16:14:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 857812992 bytes 2024-09-12T16:14:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 946923520 bytes 2024-09-12T16:14:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1093441536 bytes 2024-09-12T16:14:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1120862208 bytes 2024-09-12T16:14:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1123050496 bytes 2024-09-12T16:14:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1149704192 bytes 2024-09-12T16:14:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1207802880 bytes 2024-09-12T16:14:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1302945792 bytes 2024-09-12T16:14:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1320329216 bytes 2024-09-12T16:14:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1383746560 bytes 2024-09-12T16:14:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1419669504 bytes 2024-09-12T16:14:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1461140480 bytes 2024-09-12T16:14:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1495259136 bytes 2024-09-12T16:14:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1523243008 bytes 2024-09-12T16:14:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1546604544 bytes 2024-09-12T16:14:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1547208704 bytes 2024-09-12T16:14:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1583136768 bytes 2024-09-12T16:14:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1633282048 bytes 2024-09-12T16:14:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1647085568 bytes 2024-09-12T16:14:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1711406080 bytes 2024-09-12T16:14:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1736888320 bytes 2024-09-12T16:14:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1807885312 bytes 2024-09-12T16:14:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1827916800 bytes 2024-09-12T16:14:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1895593984 bytes 2024-09-12T16:14:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1962697728 bytes 2024-09-12T16:14:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2045896704 bytes 2024-09-12T16:14:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_50 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:14:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n snapshot ']' 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 128M ']' 2024-09-12T16:14:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=128M 2024-09-12T16:14:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:14:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ is_pool_mirror_mode_image mirror 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' image = image ']' 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:14:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-12T16:14:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:14:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_50': 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 149ede4e07d2 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.149ede4e07d2 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:14:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:49 2024 2024-09-12T16:14:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:49 2024 2024-09-12T16:14:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:49 2024 2024-09-12T16:14:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:14:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:14:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 7e4292a4-067e-49ff-9c07-bb55509086c8 2024-09-12T16:14:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:14:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n '' ']' 2024-09-12T16:14:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-12T16:14:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:14:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:14:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=9 2024-09-12T16:14:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:14:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:14:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 36249600 bytes 2024-09-12T16:14:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 92737536 bytes 2024-09-12T16:14:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 93365248 bytes 2024-09-12T16:14:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 135110656 bytes 2024-09-12T16:14:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 135737344 bytes 2024-09-12T16:14:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 181038080 bytes 2024-09-12T16:14:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 204583936 bytes 2024-09-12T16:14:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 326891520 bytes 2024-09-12T16:14:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 410104832 bytes 2024-09-12T16:14:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 445618176 bytes 2024-09-12T16:14:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 486044672 bytes 2024-09-12T16:14:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 522684416 bytes 2024-09-12T16:14:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 614066176 bytes 2024-09-12T16:14:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 646152192 bytes 2024-09-12T16:14:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 703995904 bytes 2024-09-12T16:14:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 745717760 bytes 2024-09-12T16:14:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 815676416 bytes 2024-09-12T16:14:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 834273280 bytes 2024-09-12T16:14:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 854740992 bytes 2024-09-12T16:14:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 931681280 bytes 2024-09-12T16:14:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1068203008 bytes 2024-09-12T16:14:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1087183872 bytes 2024-09-12T16:14:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1125223424 bytes 2024-09-12T16:14:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1148848128 bytes 2024-09-12T16:14:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1180538880 bytes 2024-09-12T16:14:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1197182976 bytes 2024-09-12T16:14:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1203662848 bytes 2024-09-12T16:14:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1210919936 bytes 2024-09-12T16:14:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1266741248 bytes 2024-09-12T16:14:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1353148416 bytes 2024-09-12T16:14:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1421408256 bytes 2024-09-12T16:14:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1445371904 bytes 2024-09-12T16:14:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1473715200 bytes 2024-09-12T16:14:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1485196288 bytes 2024-09-12T16:14:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1519238144 bytes 2024-09-12T16:14:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1564880896 bytes 2024-09-12T16:14:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1604400128 bytes 2024-09-12T16:14:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1635846144 bytes 2024-09-12T16:14:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1657247744 bytes 2024-09-12T16:14:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1707689984 bytes 2024-09-12T16:14:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1744147456 bytes 2024-09-12T16:14:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1744754688 bytes 2024-09-12T16:14:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1830637568 bytes 2024-09-12T16:14:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1886050304 bytes 2024-09-12T16:14:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1954939904 bytes 2024-09-12T16:14:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2007826432 bytes 2024-09-12T16:14:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2101041152 bytes 2024-09-12T16:14:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2131227648 bytes 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_count=50 2024-09-12T16:14:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:14:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local count 2024-09-12T16:14:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_count=0 2024-09-12T16:14:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:14:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 40 2024-09-12T16:14:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in `seq 1 40` 2024-09-12T16:14:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 -ne 1 2024-09-12T16:14:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-12T16:14:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep 'images: ' 2024-09-12T16:14:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cut '-d ' -f 2 2024-09-12T16:14:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T16:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ count=50 2024-09-12T16:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 50 = 50 2024-09-12T16:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-12T16:14:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:14:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state 2024-09-12T16:14:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 40 2024-09-12T16:14:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in `seq 1 40` 2024-09-12T16:14:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 -ne 1 2024-09-12T16:14:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-12T16:14:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep 'image health:' 2024-09-12T16:14:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cut '-d ' -f 3 2024-09-12T16:14:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=OK 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test OK = ERROR 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test OK = OK 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_count=50 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:14:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local count 2024-09-12T16:14:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_count=0 2024-09-12T16:14:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:14:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 40 2024-09-12T16:14:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in `seq 1 40` 2024-09-12T16:14:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 -ne 1 2024-09-12T16:14:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T16:14:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep 'images: ' 2024-09-12T16:14:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cut '-d ' -f 2 2024-09-12T16:14:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T16:14:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ count=50 2024-09-12T16:14:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 50 = 50 2024-09-12T16:14:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-12T16:14:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:14:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:14:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state 2024-09-12T16:14:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 40 2024-09-12T16:14:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in `seq 1 40` 2024-09-12T16:14:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 -ne 1 2024-09-12T16:14:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T16:14:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep 'image health:' 2024-09-12T16:14:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cut '-d ' -f 3 2024-09-12T16:14:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T16:14:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=OK 2024-09-12T16:14:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test OK = ERROR 2024-09-12T16:14:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test OK = OK 2024-09-12T16:14:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:14:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ testlog 'TEST: compare many images' 2024-09-12T16:14:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ tee -a /tmp/tmp.65UL20DnMX/rbd-mirror.test.log 2024-09-12T16:14:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ date '+%F %T' 2024-09-12T16:14:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo 2024-09-12 16:14:59 TEST: compare many images 2024-09-12T16:14:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:2024-09-12 16:14:59 TEST: compare many images 2024-09-12T16:14:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 50 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_1 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:14:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:01.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-12T16:15:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:02.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:02.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:02.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:02.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-12T16:15:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-12T16:15:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-12T16:15:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-12T16:15:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-09-12T16:15:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-09-12T16:15:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_1 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_1 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-12T16:15:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_1.status 2024-09-12T16:15:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.status 2024-09-12T16:15:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-12T16:15:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_1.status 2024-09-12T16:15:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-12T16:15:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-12T16:15:03.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 86 2024-09-12T16:15:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.status 2024-09-12T16:15:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.status 2024-09-12T16:15:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-12T16:15:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:15:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=86 2024-09-12T16:15:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:15:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-12T16:15:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=35 2024-09-12T16:15:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 86 = 35 2024-09-12T16:15:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:15:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-12T16:15:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 86 = '' 2024-09-12T16:15:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:15:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-12T16:15:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:05.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 86 = '' 2024-09-12T16:15:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:15:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.status 2024-09-12T16:15:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-12T16:15:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=86 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 86 = 86 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:15:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-12T16:15:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_1': 2024-09-12T16:15:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a56b5b6b64 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a56b5b6b64 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:06:24 2024 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:06:24 2024 2024-09-12T16:15:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:06:24 2024 2024-09-12T16:15:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:15:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:15:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:15:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 135c0a36-496a-4b49-bf12-e55a9a725020 2024-09-12T16:15:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:15:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-12T16:15:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:15:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:15:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.export 2024-09-12T16:15:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.export 2024-09-12T16:15:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.export 2024-09-12T16:15:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.export 2024-09-12T16:15:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.export 2024-09-12T16:15:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.export 2024-09-12T16:15:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_1.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_1.export 2024-09-12T16:15:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:15:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_2 2024-09-12T16:15:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-12T16:15:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-12T16:15:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-12T16:15:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-12T16:15:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-12T16:15:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-12T16:15:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-12T16:15:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-12T16:15:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-12T16:15:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-12T16:15:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_2 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_2 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-12T16:15:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_2.status 2024-09-12T16:15:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.status 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_2.status 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-12T16:15:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 88 2024-09-12T16:15:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.status 2024-09-12T16:15:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.status 2024-09-12T16:15:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-12T16:15:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:15:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=88 2024-09-12T16:15:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:15:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-12T16:15:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=36 2024-09-12T16:15:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 88 = 36 2024-09-12T16:15:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:15:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-12T16:15:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 88 = '' 2024-09-12T16:15:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:15:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-12T16:15:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 88 = '' 2024-09-12T16:15:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:15:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.status 2024-09-12T16:15:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-12T16:15:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=88 2024-09-12T16:15:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 88 = 88 2024-09-12T16:15:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-12T16:15:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:15:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-12T16:15:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_2': 2024-09-12T16:15:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:15:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:15:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:15:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5e9dd85b8 2024-09-12T16:15:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5e9dd85b8 2024-09-12T16:15:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:06:36 2024 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:06:36 2024 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:06:36 2024 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: d49cf199-0fac-4fe0-afa3-c988aa8578d1 2024-09-12T16:15:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:15:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-12T16:15:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:15:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:15:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.export 2024-09-12T16:15:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.export 2024-09-12T16:15:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.export 2024-09-12T16:15:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.export 2024-09-12T16:15:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.export 2024-09-12T16:15:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.export 2024-09-12T16:15:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_2.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_2.export 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_3 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-12T16:15:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-12T16:15:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:20.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-12T16:15:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-12T16:15:20.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-12T16:15:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-12T16:15:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-12T16:15:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-12T16:15:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-09-12T16:15:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-09-12T16:15:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_3 2024-09-12T16:15:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_3 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_3.status 2024-09-12T16:15:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.status 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_3.status 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-12T16:15:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 90 2024-09-12T16:15:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.status 2024-09-12T16:15:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.status 2024-09-12T16:15:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-12T16:15:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:15:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=90 2024-09-12T16:15:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:15:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-12T16:15:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=37 2024-09-12T16:15:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 90 = 37 2024-09-12T16:15:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:15:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-12T16:15:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 90 = '' 2024-09-12T16:15:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:15:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-12T16:15:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 90 = '' 2024-09-12T16:15:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:15:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.status 2024-09-12T16:15:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-12T16:15:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=90 2024-09-12T16:15:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 90 = 90 2024-09-12T16:15:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-12T16:15:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:15:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_3': 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a52da28b63 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a52da28b63 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:15:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:06:47 2024 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:06:47 2024 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:06:47 2024 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 50c9ce9d-ad98-4a52-8237-144bfb366358 2024-09-12T16:15:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.export 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.export 2024-09-12T16:15:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.export 2024-09-12T16:15:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.export 2024-09-12T16:15:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.export 2024-09-12T16:15:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.export 2024-09-12T16:15:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_3.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_3.export 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_4 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-12T16:15:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-12T16:15:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-12T16:15:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-12T16:15:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-12T16:15:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-12T16:15:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-12T16:15:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-12T16:15:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-12T16:15:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_4 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_4 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_4.status 2024-09-12T16:15:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.status 2024-09-12T16:15:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-12T16:15:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_4.status 2024-09-12T16:15:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.status 2024-09-12T16:15:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-12T16:15:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-12T16:15:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 92 2024-09-12T16:15:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.status 2024-09-12T16:15:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.status 2024-09-12T16:15:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-12T16:15:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:15:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=92 2024-09-12T16:15:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.status 2024-09-12T16:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.status 2024-09-12T16:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-12T16:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 92 = '' 2024-09-12T16:15:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:15:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.status 2024-09-12T16:15:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.status 2024-09-12T16:15:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-12T16:15:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 92 = '' 2024-09-12T16:15:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:15:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.status 2024-09-12T16:15:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.status 2024-09-12T16:15:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-12T16:15:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=92 2024-09-12T16:15:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 92 = 92 2024-09-12T16:15:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-12T16:15:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:15:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-12T16:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_4': 2024-09-12T16:15:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5e9ac458e 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5e9ac458e 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:02 2024 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:02 2024 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:02 2024 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:15:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:15:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 2f93870e-9268-4e8c-9b38-0d6cd6a7dd81 2024-09-12T16:15:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:15:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.export 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.export 2024-09-12T16:15:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.export 2024-09-12T16:15:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.export 2024-09-12T16:15:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.export 2024-09-12T16:15:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.export 2024-09-12T16:15:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_4.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_4.export 2024-09-12T16:15:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:15:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_5 2024-09-12T16:15:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-12T16:15:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-12T16:15:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-12T16:15:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-12T16:15:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-12T16:15:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-12T16:15:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-12T16:15:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:35.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:35.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_5 2024-09-12T16:15:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_5 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_5.status 2024-09-12T16:15:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.status 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_5.status 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-12T16:15:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 94 2024-09-12T16:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.status 2024-09-12T16:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.status 2024-09-12T16:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-12T16:15:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:15:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=94 2024-09-12T16:15:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:15:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-12T16:15:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=39 2024-09-12T16:15:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 94 = 39 2024-09-12T16:15:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:15:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-12T16:15:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 94 = '' 2024-09-12T16:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:15:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-12T16:15:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 94 = '' 2024-09-12T16:15:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:15:38.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7642571699074905) > weight (0.25) 2024-09-12T16:15:38.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.619094512576682) > weight (0.25) 2024-09-12T16:15:38.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9505951405217775) > weight (0.25) 2024-09-12T16:15:38.405 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8354714276724674) > weight (0.25) 2024-09-12T16:15:38.405 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-12T16:15:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.status 2024-09-12T16:15:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-12T16:15:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=94 2024-09-12T16:15:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 94 = 94 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:15:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-12T16:15:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_5': 2024-09-12T16:15:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:15:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:15:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:15:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5bc7068a2 2024-09-12T16:15:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5bc7068a2 2024-09-12T16:15:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:17 2024 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:17 2024 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:17 2024 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 4e25a06d-cee5-4e43-8e00-5e4bc8c2f14a 2024-09-12T16:15:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:15:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-12T16:15:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:15:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:15:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.export 2024-09-12T16:15:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.export 2024-09-12T16:15:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.export 2024-09-12T16:15:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.export 2024-09-12T16:15:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.export 2024-09-12T16:15:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.export 2024-09-12T16:15:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_5.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_5.export 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_6 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-12T16:15:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-12T16:15:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-12T16:15:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-12T16:15:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-12T16:15:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-12T16:15:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:44.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-12T16:15:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-12T16:15:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-09-12T16:15:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-09-12T16:15:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_6 2024-09-12T16:15:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:44.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_6 2024-09-12T16:15:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-12T16:15:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_6.status 2024-09-12T16:15:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.status 2024-09-12T16:15:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-12T16:15:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_6.status 2024-09-12T16:15:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-12T16:15:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-12T16:15:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 96 2024-09-12T16:15:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.status 2024-09-12T16:15:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.status 2024-09-12T16:15:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-12T16:15:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:15:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=96 2024-09-12T16:15:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:15:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-12T16:15:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=40 2024-09-12T16:15:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 96 = 40 2024-09-12T16:15:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:15:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-12T16:15:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 96 = '' 2024-09-12T16:15:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:46.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:15:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-12T16:15:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 96 = '' 2024-09-12T16:15:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:15:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.status 2024-09-12T16:15:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-12T16:15:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=96 2024-09-12T16:15:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 96 = 96 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:15:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_6': 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a54097a4a3 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a54097a4a3 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:15:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:25 2024 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:25 2024 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:25 2024 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 7116ac0d-b88f-466e-8403-60e8e5d9a276 2024-09-12T16:15:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.export 2024-09-12T16:15:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.export 2024-09-12T16:15:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.export 2024-09-12T16:15:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.export 2024-09-12T16:15:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.export 2024-09-12T16:15:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.export 2024-09-12T16:15:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_6.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_6.export 2024-09-12T16:15:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_7 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-12T16:15:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-12T16:15:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-12T16:15:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-12T16:15:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-12T16:15:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-12T16:15:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-12T16:15:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-12T16:15:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-09-12T16:15:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_7 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_7 2024-09-12T16:15:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-12T16:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_7.status 2024-09-12T16:15:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.status 2024-09-12T16:15:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-12T16:15:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_7.status 2024-09-12T16:15:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.status 2024-09-12T16:15:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-12T16:15:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-12T16:15:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 98 2024-09-12T16:15:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.status 2024-09-12T16:15:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.status 2024-09-12T16:15:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-12T16:15:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:15:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=98 2024-09-12T16:15:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:15:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.status 2024-09-12T16:15:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.status 2024-09-12T16:15:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-12T16:15:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:15:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:15:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:15:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:15:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:15:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 98 = '' 2024-09-12T16:15:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:15:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:15:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.status 2024-09-12T16:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.status 2024-09-12T16:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-12T16:15:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:15:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:15:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=98 2024-09-12T16:15:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 98 = 98 2024-09-12T16:15:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-12T16:15:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:15:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_7': 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5d75fae2 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5d75fae2 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:15:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:35 2024 2024-09-12T16:15:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:35 2024 2024-09-12T16:15:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:35 2024 2024-09-12T16:15:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:15:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:15:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:15:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 9ed94e20-8486-411e-a66b-35ec4ef5bf45 2024-09-12T16:15:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.export 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.export 2024-09-12T16:15:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.export 2024-09-12T16:15:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.export 2024-09-12T16:15:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:56.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.export 2024-09-12T16:15:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:15:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.export 2024-09-12T16:15:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_7.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_7.export 2024-09-12T16:15:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:15:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_8 2024-09-12T16:15:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-12T16:15:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:15:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:15:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:15:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-12T16:15:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:15:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-12T16:15:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:15:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:15:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:15:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:15:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:15:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:15:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-12T16:15:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:15:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:15:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:15:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:15:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:15:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:15:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:15:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:15:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-12T16:15:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-12T16:15:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:15:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:15:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:15:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:15:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:15:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:15:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:15:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-12T16:15:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-12T16:15:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:15:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:15:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:15:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:15:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:15:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:15:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-09-12T16:15:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:15:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:15:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:15:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:15:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:15:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_8 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_8 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:15:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-12T16:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_8.status 2024-09-12T16:15:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.status 2024-09-12T16:15:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-12T16:15:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_8.status 2024-09-12T16:15:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:15:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:15:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-12T16:15:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:15:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:15:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:15:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-12T16:16:00.392 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9090942287302693) > weight (0.25) 2024-09-12T16:16:00.392 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7014806542989372) > weight (0.25) 2024-09-12T16:16:00.392 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-12T16:16:00.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-12T16:16:00.393 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-12T16:16:00.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:16:00.393+0000 7f57cef07640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 38100) UID: 1000 2024-09-12T16:16:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 100 2024-09-12T16:16:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.status 2024-09-12T16:16:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.status 2024-09-12T16:16:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-12T16:16:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=100 2024-09-12T16:16:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-12T16:16:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=42 2024-09-12T16:16:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 100 = 42 2024-09-12T16:16:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:16:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-12T16:16:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=42 2024-09-12T16:16:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 100 = 42 2024-09-12T16:16:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:16:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-12T16:16:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=42 2024-09-12T16:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 100 = 42 2024-09-12T16:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:16:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-12T16:16:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=42 2024-09-12T16:16:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 100 = 42 2024-09-12T16:16:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 2 2024-09-12T16:16:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-12T16:16:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 100 = '' 2024-09-12T16:16:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 2 2024-09-12T16:16:06.394 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-12T16:16:06.395 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:16:06.395 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-12T16:16:06.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-12T16:16:06.395 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-12T16:16:06.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-12T16:16:06.399 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-12T16:16:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.status 2024-09-12T16:16:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-12T16:16:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=100 2024-09-12T16:16:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 100 = 100 2024-09-12T16:16:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-12T16:16:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:16:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-12T16:16:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_8': 2024-09-12T16:16:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5799adbf4 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5799adbf4 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:40 2024 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:40 2024 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:40 2024 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:16:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:16:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:16:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 7246287d-3dc0-422d-9d93-915c9d37781e 2024-09-12T16:16:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:16:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.export 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.export 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.export 2024-09-12T16:16:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.export 2024-09-12T16:16:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.export 2024-09-12T16:16:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.export 2024-09-12T16:16:09.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_8.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_8.export 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_9 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:16:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-12T16:16:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:16:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:16:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:16:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:16:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:16:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:16:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:16:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:16:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:16:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-12T16:16:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-12T16:16:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:16:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:16:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:16:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:16:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-12T16:16:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-12T16:16:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:16:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2024-09-12T16:16:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-12T16:16:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_9 2024-09-12T16:16:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:16:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:16:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:16:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:16:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_9 2024-09-12T16:16:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:16:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:16:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:16:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:16:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:16:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:16:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_9 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_9 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-12T16:16:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_9.status 2024-09-12T16:16:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.status 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_9.status 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-12T16:16:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 102 2024-09-12T16:16:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.status 2024-09-12T16:16:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.status 2024-09-12T16:16:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-12T16:16:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=102 2024-09-12T16:16:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-12T16:16:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=43 2024-09-12T16:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 102 = 43 2024-09-12T16:16:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:16:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-12T16:16:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 102 = '' 2024-09-12T16:16:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:16:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-12T16:16:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 102 = '' 2024-09-12T16:16:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.status 2024-09-12T16:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-12T16:16:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=102 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 102 = 102 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:16:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-12T16:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_9': 2024-09-12T16:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a58971d5c0 2024-09-12T16:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a58971d5c0 2024-09-12T16:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:07:51 2024 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:07:51 2024 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:07:51 2024 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: f574749e-1611-4718-9f27-0329fe616606 2024-09-12T16:16:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:16:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-12T16:16:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:16:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:16:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.export 2024-09-12T16:16:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.export 2024-09-12T16:16:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.export 2024-09-12T16:16:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.export 2024-09-12T16:16:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.export 2024-09-12T16:16:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.export 2024-09-12T16:16:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_9.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_9.export 2024-09-12T16:16:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:16:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_10 2024-09-12T16:16:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-12T16:16:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:16:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:16:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-12T16:16:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:16:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:16:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:16:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:16:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:16:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:16:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:16:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:16:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-12T16:16:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-12T16:16:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:16:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:16:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:16:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-12T16:16:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-12T16:16:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:16:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:16:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:20.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:16:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:16:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:16:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-09-12T16:16:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-12T16:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-09-12T16:16:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:16:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:16:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:16:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:16:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-09-12T16:16:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:16:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:16:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:16:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:16:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_10 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_10 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_10.status 2024-09-12T16:16:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.status 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_10.status 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-12T16:16:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 104 2024-09-12T16:16:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.status 2024-09-12T16:16:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.status 2024-09-12T16:16:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-12T16:16:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=104 2024-09-12T16:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-12T16:16:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=44 2024-09-12T16:16:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 104 = 44 2024-09-12T16:16:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-12T16:16:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 104 = '' 2024-09-12T16:16:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:16:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-12T16:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 104 = '' 2024-09-12T16:16:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:16:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.status 2024-09-12T16:16:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-12T16:16:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=104 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 104 = 104 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:16:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_10': 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a597bf74d9 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a597bf74d9 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:16:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:03 2024 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:03 2024 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:03 2024 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c558b2dc-a46e-4a16-bc7f-98af17a10d37 2024-09-12T16:16:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:16:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-12T16:16:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:16:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:16:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.export 2024-09-12T16:16:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.export 2024-09-12T16:16:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.export 2024-09-12T16:16:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.export 2024-09-12T16:16:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.export 2024-09-12T16:16:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.export 2024-09-12T16:16:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_10.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_10.export 2024-09-12T16:16:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:16:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_11 2024-09-12T16:16:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-12T16:16:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:16:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-12T16:16:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-12T16:16:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:16:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:16:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-12T16:16:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:16:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:16:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:16:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:16:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:16:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:16:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:16:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:16:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:16:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:16:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-12T16:16:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-12T16:16:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-12T16:16:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:16:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:16:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:16:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:16:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-12T16:16:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-12T16:16:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-09-12T16:16:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:16:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:16:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:16:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:16:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:16:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-09-12T16:16:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-12T16:16:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-09-12T16:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:16:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:16:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:16:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-09-12T16:16:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:16:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:16:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:16:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:16:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:16:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_11 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:16:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_11 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_11.status 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.status 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-12T16:16:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_11.status 2024-09-12T16:16:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-12T16:16:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-12T16:16:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 106 2024-09-12T16:16:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.status 2024-09-12T16:16:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.status 2024-09-12T16:16:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-12T16:16:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=106 2024-09-12T16:16:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-12T16:16:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=45 2024-09-12T16:16:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 106 = 45 2024-09-12T16:16:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:16:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-12T16:16:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:30.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 106 = '' 2024-09-12T16:16:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:16:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-12T16:16:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 106 = '' 2024-09-12T16:16:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:30.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:16:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.status 2024-09-12T16:16:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-12T16:16:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=106 2024-09-12T16:16:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 106 = 106 2024-09-12T16:16:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-12T16:16:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:16:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_11': 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5b98ebe6d 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5b98ebe6d 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:16:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:11 2024 2024-09-12T16:16:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:11 2024 2024-09-12T16:16:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:11 2024 2024-09-12T16:16:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:16:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:16:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:16:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c7fac33d-e73d-4d4b-8daf-469b5cf5e4e4 2024-09-12T16:16:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:16:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-12T16:16:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:16:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:16:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.export 2024-09-12T16:16:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.export 2024-09-12T16:16:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.export 2024-09-12T16:16:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.export 2024-09-12T16:16:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.export 2024-09-12T16:16:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.export 2024-09-12T16:16:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_11.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_11.export 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_12 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:16:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-12T16:16:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:36.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-12T16:16:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:16:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:16:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:16:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-12T16:16:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:16:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:16:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:16:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:16:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:16:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:16:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:16:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:16:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:16:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:16:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:16:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:16:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-12T16:16:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-12T16:16:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:16:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:16:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:16:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-12T16:16:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-12T16:16:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:16:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:16:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:16:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-09-12T16:16:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-12T16:16:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2024-09-12T16:16:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:16:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:16:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:16:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:16:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:16:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2024-09-12T16:16:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:16:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:16:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:16:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:16:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_12 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:16:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_12 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_12.status 2024-09-12T16:16:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.status 2024-09-12T16:16:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-12T16:16:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_12.status 2024-09-12T16:16:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-12T16:16:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-12T16:16:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 108 2024-09-12T16:16:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.status 2024-09-12T16:16:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.status 2024-09-12T16:16:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-12T16:16:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=108 2024-09-12T16:16:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-12T16:16:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=46 2024-09-12T16:16:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 108 = 46 2024-09-12T16:16:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:16:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-12T16:16:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 108 = '' 2024-09-12T16:16:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:16:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-12T16:16:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 108 = '' 2024-09-12T16:16:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:38.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.status 2024-09-12T16:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-12T16:16:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=108 2024-09-12T16:16:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 108 = 108 2024-09-12T16:16:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-12T16:16:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:16:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-12T16:16:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_12': 2024-09-12T16:16:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:16:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5e7930169 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5e7930169 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:23 2024 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:23 2024 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:23 2024 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c928224c-98a5-45be-8f56-7a1ae44aa427 2024-09-12T16:16:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:16:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-12T16:16:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:16:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:16:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.export 2024-09-12T16:16:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.export 2024-09-12T16:16:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.export 2024-09-12T16:16:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.export 2024-09-12T16:16:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.export 2024-09-12T16:16:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.export 2024-09-12T16:16:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_12.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_12.export 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_13 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:16:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-12T16:16:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-12T16:16:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:16:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:16:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:16:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:16:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:16:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-12T16:16:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:16:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:16:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:16:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:16:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:16:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-12T16:16:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-12T16:16:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-12T16:16:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:16:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:16:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:16:44.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-12T16:16:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-12T16:16:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:16:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:16:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:16:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-09-12T16:16:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-12T16:16:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:16:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:16:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2024-09-12T16:16:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:16:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:16:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:16:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:16:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_13 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_13 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_13.status 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.status 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_13.status 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.status 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-12T16:16:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 110 2024-09-12T16:16:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.status 2024-09-12T16:16:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.status 2024-09-12T16:16:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-12T16:16:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=110 2024-09-12T16:16:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.status 2024-09-12T16:16:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.status 2024-09-12T16:16:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-12T16:16:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:45.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=47 2024-09-12T16:16:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 110 = 47 2024-09-12T16:16:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.status 2024-09-12T16:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.status 2024-09-12T16:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-12T16:16:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 110 = '' 2024-09-12T16:16:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:16:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.status 2024-09-12T16:16:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.status 2024-09-12T16:16:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-12T16:16:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=110 2024-09-12T16:16:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 110 = 110 2024-09-12T16:16:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:16:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_13': 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5480d52fa 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5480d52fa 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:36 2024 2024-09-12T16:16:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:36 2024 2024-09-12T16:16:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:36 2024 2024-09-12T16:16:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:16:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:16:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:16:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 02a2eb1f-5638-48d4-814e-2b3a49935a89 2024-09-12T16:16:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:16:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.export 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.export 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.export 2024-09-12T16:16:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.export 2024-09-12T16:16:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.export 2024-09-12T16:16:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.export 2024-09-12T16:16:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_13.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_13.export 2024-09-12T16:16:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:16:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_14 2024-09-12T16:16:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-12T16:16:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:16:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-12T16:16:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-12T16:16:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:16:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:16:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:16:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-12T16:16:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:16:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:16:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:16:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:16:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:16:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:16:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:16:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:16:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:16:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:16:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-12T16:16:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-12T16:16:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:51.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:51.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:51.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:16:51.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:16:51.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:16:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-12T16:16:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-12T16:16:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:16:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:16:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:16:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-09-12T16:16:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-12T16:16:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:16:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:16:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:16:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-09-12T16:16:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:16:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:16:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:16:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:16:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_14 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_14 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_14.status 2024-09-12T16:16:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.status 2024-09-12T16:16:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_14.status 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.status 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-12T16:16:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 112 2024-09-12T16:16:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.status 2024-09-12T16:16:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.status 2024-09-12T16:16:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-12T16:16:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=112 2024-09-12T16:16:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.status 2024-09-12T16:16:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.status 2024-09-12T16:16:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-12T16:16:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=48 2024-09-12T16:16:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 112 = 48 2024-09-12T16:16:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:16:53.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.status 2024-09-12T16:16:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.status 2024-09-12T16:16:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-12T16:16:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 112 = '' 2024-09-12T16:16:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.status 2024-09-12T16:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.status 2024-09-12T16:16:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-12T16:16:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=112 2024-09-12T16:16:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 112 = 112 2024-09-12T16:16:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-12T16:16:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:16:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_14': 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5ac10b890 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5ac10b890 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:46 2024 2024-09-12T16:16:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:46 2024 2024-09-12T16:16:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:46 2024 2024-09-12T16:16:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:16:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:16:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:16:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 4d0d2df1-b0b8-4abb-b611-0cc3a70369d4 2024-09-12T16:16:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.export 2024-09-12T16:16:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.export 2024-09-12T16:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.export 2024-09-12T16:16:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.export 2024-09-12T16:16:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.export 2024-09-12T16:16:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:16:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.export 2024-09-12T16:16:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_14.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_14.export 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_15 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:16:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:16:57.501 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.500+0000 7fbc5a793640 -1 reset not still connected to 0x55f82c87fba0 2024-09-12T16:16:57.509 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.507+0000 7fbc5a793640 -1 reset not still connected to 0x55f830c98820 2024-09-12T16:16:57.510 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.508+0000 7fbc5a793640 -1 reset not still connected to 0x55f830c99790 2024-09-12T16:16:57.510 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.508+0000 7fbc5a793640 -1 reset not still connected to 0x55f830cc1930 2024-09-12T16:16:57.510 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.508+0000 7fbc5a793640 -1 reset not still connected to 0x55f839559520 2024-09-12T16:16:57.513 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.510+0000 7fbc5a793640 -1 reset not still connected to 0x55f83a610750 2024-09-12T16:16:57.513 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.510+0000 7fbc5a793640 -1 reset not still connected to 0x55f83a6109c0 2024-09-12T16:16:57.513 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.510+0000 7fbc5a793640 -1 reset not still connected to 0x55f83a610a90 2024-09-12T16:16:57.514 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f83a611790 2024-09-12T16:16:57.514 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f841b8a750 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f841b8bc70 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f85b9fc270 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f85b9fcd00 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f85beb81a0 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f85e702000 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f85e702b60 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f860ea3c70 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f862754410 2024-09-12T16:16:57.515 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:16:57.513+0000 7fbc5a793640 -1 reset not still connected to 0x55f862755a00 2024-09-12T16:16:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-12T16:16:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:16:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-12T16:16:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:16:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:16:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:16:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:16:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:16:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:16:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:16:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:16:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:16:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:16:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:16:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:16:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-12T16:16:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-12T16:16:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:16:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:16:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:16:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-12T16:16:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-12T16:16:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-09-12T16:16:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:16:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:16:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:16:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:16:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:16:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-09-12T16:16:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-12T16:16:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:16:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:16:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:16:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_15 2024-09-12T16:16:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:16:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:16:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:16:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:16:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:16:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:16:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_15 2024-09-12T16:16:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:16:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:16:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:16:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:16:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:16:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:16:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:16:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:16:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_15 2024-09-12T16:16:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_15 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-12T16:16:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_15.status 2024-09-12T16:16:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.status 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_15.status 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.status 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-12T16:16:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 114 2024-09-12T16:16:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.status 2024-09-12T16:16:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:16:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.status 2024-09-12T16:16:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-12T16:16:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:16:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=114 2024-09-12T16:16:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:16:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.status 2024-09-12T16:16:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:16:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:16:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:16:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.status 2024-09-12T16:16:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-12T16:16:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:16:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:16:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:16:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:16:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:16:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:16:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:16:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 114 = '' 2024-09-12T16:16:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:16:59.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.status 2024-09-12T16:17:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:17:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.status 2024-09-12T16:17:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-12T16:17:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 114 = '' 2024-09-12T16:17:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.status 2024-09-12T16:17:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:17:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.status 2024-09-12T16:17:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-12T16:17:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=114 2024-09-12T16:17:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 114 = 114 2024-09-12T16:17:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-12T16:17:01.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:17:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_15': 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5b860dc4e 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5b860dc4e 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:08:54 2024 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:08:54 2024 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:08:54 2024 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 1961696a-f14e-44a0-a222-ad6e28c429fe 2024-09-12T16:17:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.export 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.export 2024-09-12T16:17:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.export 2024-09-12T16:17:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.export 2024-09-12T16:17:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.export 2024-09-12T16:17:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.export 2024-09-12T16:17:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_15.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_15.export 2024-09-12T16:17:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_16 2024-09-12T16:17:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-12T16:17:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:17:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-12T16:17:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:17:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:17:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:17:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:17:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:17:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:17:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:17:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:17:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:17:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-12T16:17:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-12T16:17:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:17:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:17:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:17:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-12T16:17:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-12T16:17:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-12T16:17:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:17:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:17:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:17:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:17:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-12T16:17:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-12T16:17:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_16 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:17:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:17:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:17:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:17:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_16 2024-09-12T16:17:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:17:05.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:17:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:17:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:17:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:17:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:17:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_16 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_16 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_16.status 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.status 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_16.status 2024-09-12T16:17:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.status 2024-09-12T16:17:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-12T16:17:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-12T16:17:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 116 2024-09-12T16:17:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.status 2024-09-12T16:17:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.status 2024-09-12T16:17:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-12T16:17:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:17:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=116 2024-09-12T16:17:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:17:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.status 2024-09-12T16:17:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.status 2024-09-12T16:17:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-12T16:17:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 116 = '' 2024-09-12T16:17:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.status 2024-09-12T16:17:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.status 2024-09-12T16:17:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-12T16:17:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 116 = '' 2024-09-12T16:17:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.status 2024-09-12T16:17:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.status 2024-09-12T16:17:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-12T16:17:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=116 2024-09-12T16:17:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 116 = 116 2024-09-12T16:17:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-12T16:17:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_16': 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a55a9bf0b 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a55a9bf0b 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:09 2024 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:09 2024 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:09 2024 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 47d38d69-792b-405f-a65b-101acd193fb9 2024-09-12T16:17:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-12T16:17:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:17:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.export 2024-09-12T16:17:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.export 2024-09-12T16:17:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.export 2024-09-12T16:17:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.export 2024-09-12T16:17:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.export 2024-09-12T16:17:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.export 2024-09-12T16:17:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_16.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_16.export 2024-09-12T16:17:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_17 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:17:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-12T16:17:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:17:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:17:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:17:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-12T16:17:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:17:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:17:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:17:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:17:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:17:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:17:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:17:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:17:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-12T16:17:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:17:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:17:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:17:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:17:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-12T16:17:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-12T16:17:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:17:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:17:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:17:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:17:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-12T16:17:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-12T16:17:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:17:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:17:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:17:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-09-12T16:17:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-12T16:17:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2024-09-12T16:17:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:17:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:17:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:17:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:17:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2024-09-12T16:17:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:17:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:17:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:17:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:17:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:17:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:17:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:17:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_17 2024-09-12T16:17:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_17 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_17.status 2024-09-12T16:17:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.status 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_17.status 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.status 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-12T16:17:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 118 2024-09-12T16:17:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.status 2024-09-12T16:17:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.status 2024-09-12T16:17:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-12T16:17:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:17:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=118 2024-09-12T16:17:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:17:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.status 2024-09-12T16:17:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.status 2024-09-12T16:17:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-12T16:17:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 118 = '' 2024-09-12T16:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.status 2024-09-12T16:17:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.status 2024-09-12T16:17:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-12T16:17:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 118 = '' 2024-09-12T16:17:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.status 2024-09-12T16:17:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.status 2024-09-12T16:17:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-12T16:17:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=118 2024-09-12T16:17:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 118 = 118 2024-09-12T16:17:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:15.864 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.861+0000 7fae4b5c9640 -1 reset not still connected to 0x55c4fe8184e0 2024-09-12T16:17:15.864 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.861+0000 7fae4b5c9640 -1 reset not still connected to 0x55c4fe819450 2024-09-12T16:17:15.869 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.868+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5001ba270 2024-09-12T16:17:15.869 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.868+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5001ba9c0 2024-09-12T16:17:15.874 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c500608820 2024-09-12T16:17:15.874 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5006096c0 2024-09-12T16:17:15.874 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5008f6680 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5008f6b60 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5032664e0 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c503266680 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c503267450 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c505964d00 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5059652b0 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c505965450 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5059655f0 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5060964e0 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5060968f0 2024-09-12T16:17:15.875 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c506096b60 2024-09-12T16:17:15.876 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c506096f70 2024-09-12T16:17:15.876 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c506097110 2024-09-12T16:17:15.876 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5060971e0 2024-09-12T16:17:15.876 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:17:15.873+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5067f1110 2024-09-12T16:17:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_17': 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a51f83d072 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a51f83d072 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:23 2024 2024-09-12T16:17:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:23 2024 2024-09-12T16:17:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:23 2024 2024-09-12T16:17:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 3d399251-fa2f-40f2-9b35-dfd5e3c5d326 2024-09-12T16:17:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.export 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.export 2024-09-12T16:17:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.export 2024-09-12T16:17:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.export 2024-09-12T16:17:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.export 2024-09-12T16:17:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.export 2024-09-12T16:17:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_17.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_17.export 2024-09-12T16:17:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_18 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:17:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-12T16:17:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:17:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:17:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:17:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:17:18.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:17:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-12T16:17:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:17:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:18.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:17:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:17:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:17:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:17:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:17:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-12T16:17:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-12T16:17:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-12T16:17:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:17:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:17:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:17:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:17:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-12T16:17:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-12T16:17:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:17:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:17:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:17:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-09-12T16:17:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-12T16:17:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2024-09-12T16:17:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:17:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:17:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:17:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:17:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2024-09-12T16:17:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:17:19.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:17:19.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:17:19.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:17:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:17:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:17:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:17:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_18 2024-09-12T16:17:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_18 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-12T16:17:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_18.status 2024-09-12T16:17:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.status 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_18.status 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-12T16:17:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 120 2024-09-12T16:17:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.status 2024-09-12T16:17:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.status 2024-09-12T16:17:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-12T16:17:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:17:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=120 2024-09-12T16:17:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:17:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-12T16:17:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=52 2024-09-12T16:17:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 120 = 52 2024-09-12T16:17:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-12T16:17:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=52 2024-09-12T16:17:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 120 = 52 2024-09-12T16:17:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-12T16:17:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 120 = '' 2024-09-12T16:17:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:17:21.977 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:17:21.975+0000 7f765f9cc640 -1 reset not still connected to 0x55d0c5129ba0 2024-09-12T16:17:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.status 2024-09-12T16:17:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-12T16:17:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=120 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 120 = 120 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_18': 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a54b611fdb 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a54b611fdb 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:29 2024 2024-09-12T16:17:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:29 2024 2024-09-12T16:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:29 2024 2024-09-12T16:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 77378832-95d1-4af4-a433-6685e1d5dd58 2024-09-12T16:17:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.export 2024-09-12T16:17:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.export 2024-09-12T16:17:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.export 2024-09-12T16:17:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.export 2024-09-12T16:17:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.export 2024-09-12T16:17:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.export 2024-09-12T16:17:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_18.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_18.export 2024-09-12T16:17:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_19 2024-09-12T16:17:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-12T16:17:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:17:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-12T16:17:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:17:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:17:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:17:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-12T16:17:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:17:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:17:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:17:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:17:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:17:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:17:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:17:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:17:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:17:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-12T16:17:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-12T16:17:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-09-12T16:17:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:17:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:17:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:17:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:17:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-09-12T16:17:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-12T16:17:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:17:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:17:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:17:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2024-09-12T16:17:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-12T16:17:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_19 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:17:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:17:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:17:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_19 2024-09-12T16:17:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:17:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:17:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:17:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:17:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:17:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_19 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:17:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_19 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_19.status 2024-09-12T16:17:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.status 2024-09-12T16:17:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-12T16:17:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_19.status 2024-09-12T16:17:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-12T16:17:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-12T16:17:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 122 2024-09-12T16:17:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.status 2024-09-12T16:17:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.status 2024-09-12T16:17:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-12T16:17:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:17:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=122 2024-09-12T16:17:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:17:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-12T16:17:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=53 2024-09-12T16:17:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 122 = 53 2024-09-12T16:17:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-12T16:17:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=53 2024-09-12T16:17:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 122 = 53 2024-09-12T16:17:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:29.407 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.585882063903688) > weight (0.25) 2024-09-12T16:17:29.407 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7557989742207972) > weight (0.25) 2024-09-12T16:17:29.407 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9784658872751818) > weight (0.25) 2024-09-12T16:17:29.407 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-12T16:17:29.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-12T16:17:29.408 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-12T16:17:29.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:17:29.407+0000 7f7596fa5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 38197) UID: 1000 2024-09-12T16:17:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-12T16:17:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 122 = '' 2024-09-12T16:17:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:17:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.status 2024-09-12T16:17:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-12T16:17:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=122 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 122 = 122 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-12T16:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_19': 2024-09-12T16:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a57992d6d6 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a57992d6d6 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:35 2024 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:35 2024 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:35 2024 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 2152ea11-154f-4401-9170-49378499a08c 2024-09-12T16:17:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.export 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.export 2024-09-12T16:17:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.export 2024-09-12T16:17:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.export 2024-09-12T16:17:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.export 2024-09-12T16:17:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.export 2024-09-12T16:17:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_19.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_19.export 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_20 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-12T16:17:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:17:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:17:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:17:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-12T16:17:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:17:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:17:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:17:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:17:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:17:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:17:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:17:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:17:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:17:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-12T16:17:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-12T16:17:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:17:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:17:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-12T16:17:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-12T16:17:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:17:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:17:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:17:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-09-12T16:17:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-12T16:17:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:17:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:17:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-09-12T16:17:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:17:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:17:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:17:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:17:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:17:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:17:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_20 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_20 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-12T16:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_20.status 2024-09-12T16:17:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.status 2024-09-12T16:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-12T16:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_20.status 2024-09-12T16:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-12T16:17:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-12T16:17:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 124 2024-09-12T16:17:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.status 2024-09-12T16:17:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.status 2024-09-12T16:17:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-12T16:17:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:17:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=124 2024-09-12T16:17:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:17:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-12T16:17:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=54 2024-09-12T16:17:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 124 = 54 2024-09-12T16:17:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-12T16:17:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 124 = '' 2024-09-12T16:17:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-12T16:17:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 124 = '' 2024-09-12T16:17:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:17:38.409 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-12T16:17:38.409 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:17:38.409 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-12T16:17:38.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-12T16:17:38.410 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-12T16:17:38.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-12T16:17:38.413 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-12T16:17:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.status 2024-09-12T16:17:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-12T16:17:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=124 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 124 = 124 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_20': 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5566b785d 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5566b785d 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:42 2024 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:42 2024 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:42 2024 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: ac9510df-01cf-4e79-97cb-6868ae398ef5 2024-09-12T16:17:40.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-12T16:17:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:17:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.export 2024-09-12T16:17:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.export 2024-09-12T16:17:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.export 2024-09-12T16:17:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.export 2024-09-12T16:17:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.export 2024-09-12T16:17:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.export 2024-09-12T16:17:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_20.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_20.export 2024-09-12T16:17:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_21 2024-09-12T16:17:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-12T16:17:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:17:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-12T16:17:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-12T16:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:17:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-12T16:17:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:17:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:17:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:17:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:17:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:17:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:17:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:17:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:17:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:17:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:17:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:17:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:17:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-12T16:17:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-12T16:17:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-09-12T16:17:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:17:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-09-12T16:17:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-12T16:17:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-09-12T16:17:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:17:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:17:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:17:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:17:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-09-12T16:17:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-12T16:17:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_21 2024-09-12T16:17:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:17:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:17:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:17:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:17:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:17:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_21 2024-09-12T16:17:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:17:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:17:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:17:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:17:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:17:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:17:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_21 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_21 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-12T16:17:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_21.status 2024-09-12T16:17:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.status 2024-09-12T16:17:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-12T16:17:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_21.status 2024-09-12T16:17:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-12T16:17:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-12T16:17:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 126 2024-09-12T16:17:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.status 2024-09-12T16:17:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.status 2024-09-12T16:17:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-12T16:17:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:17:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=126 2024-09-12T16:17:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:17:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-12T16:17:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=55 2024-09-12T16:17:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 126 = 55 2024-09-12T16:17:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-12T16:17:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 126 = '' 2024-09-12T16:17:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-12T16:17:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 126 = '' 2024-09-12T16:17:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:17:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.status 2024-09-12T16:17:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-12T16:17:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=126 2024-09-12T16:17:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 126 = 126 2024-09-12T16:17:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-12T16:17:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-12T16:17:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_21': 2024-09-12T16:17:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a519e3f9d9 2024-09-12T16:17:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a519e3f9d9 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:09:54 2024 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:09:54 2024 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:09:54 2024 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a8bf7886-b7c1-43a8-820b-159e65038b93 2024-09-12T16:17:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-12T16:17:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:17:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.export 2024-09-12T16:17:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.export 2024-09-12T16:17:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.export 2024-09-12T16:17:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.export 2024-09-12T16:17:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.export 2024-09-12T16:17:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.export 2024-09-12T16:17:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_21.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_21.export 2024-09-12T16:17:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_22 2024-09-12T16:17:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-12T16:17:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:17:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-12T16:17:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-12T16:17:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:17:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:17:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:17:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-12T16:17:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:17:53.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:17:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:17:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:17:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:17:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:17:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:17:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:17:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:17:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:17:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:17:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:17:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:17:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-12T16:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-12T16:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-12T16:17:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:17:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:17:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:17:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:17:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-12T16:17:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-12T16:17:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-09-12T16:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:17:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:17:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-09-12T16:17:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-12T16:17:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:17:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:17:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:17:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:17:53.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:17:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:17:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:17:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:17:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:17:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:17:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2024-09-12T16:17:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:17:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:17:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:17:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:17:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:17:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:17:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_22 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_22 2024-09-12T16:17:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-12T16:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_22.status 2024-09-12T16:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.status 2024-09-12T16:17:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-12T16:17:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_22.status 2024-09-12T16:17:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:17:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-12T16:17:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-12T16:17:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 128 2024-09-12T16:17:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.status 2024-09-12T16:17:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.status 2024-09-12T16:17:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-12T16:17:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:17:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=128 2024-09-12T16:17:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:17:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-12T16:17:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=56 2024-09-12T16:17:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 128 = 56 2024-09-12T16:17:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:17:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-12T16:17:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 128 = '' 2024-09-12T16:17:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-12T16:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:17:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:17:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:17:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:17:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:17:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 128 = '' 2024-09-12T16:17:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:17:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.status 2024-09-12T16:17:57.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:17:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-12T16:17:57.589 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:17:57.588+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b5a23a00 2024-09-12T16:17:57.589 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:17:57.588+0000 7f765f9cc640 -1 reset not still connected to 0x55d0bb6dfad0 2024-09-12T16:17:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:17:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=128 2024-09-12T16:17:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 128 = 128 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:17:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:17:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-12T16:17:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_22': 2024-09-12T16:17:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5dc8f944e 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5dc8f944e 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:09 2024 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:09 2024 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:09 2024 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:17:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:17:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 051e4edc-7365-4ada-b234-2ea4450f9d6d 2024-09-12T16:17:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:17:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.export 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.export 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.export 2024-09-12T16:17:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.export 2024-09-12T16:17:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.export 2024-09-12T16:17:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:17:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.export 2024-09-12T16:17:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_22.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_22.export 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_23 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:17:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-12T16:18:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-12T16:18:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-12T16:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-12T16:18:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-12T16:18:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-09-12T16:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-12T16:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:01.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2024-09-12T16:18:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-12T16:18:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2024-09-12T16:18:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_23 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_23 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_23.status 2024-09-12T16:18:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.status 2024-09-12T16:18:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-12T16:18:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_23.status 2024-09-12T16:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.status 2024-09-12T16:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-12T16:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-12T16:18:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 130 2024-09-12T16:18:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.status 2024-09-12T16:18:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.status 2024-09-12T16:18:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-12T16:18:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=130 2024-09-12T16:18:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.status 2024-09-12T16:18:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.status 2024-09-12T16:18:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-12T16:18:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=57 2024-09-12T16:18:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 130 = 57 2024-09-12T16:18:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.status 2024-09-12T16:18:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.status 2024-09-12T16:18:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-12T16:18:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:03.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 130 = '' 2024-09-12T16:18:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.status 2024-09-12T16:18:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.status 2024-09-12T16:18:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-12T16:18:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=130 2024-09-12T16:18:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 130 = 130 2024-09-12T16:18:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:04.400 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6466786366903317) > weight (0.25) 2024-09-12T16:18:04.400 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-12T16:18:04.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-12T16:18:04.401 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-12T16:18:04.401 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-12T16:18:04.401 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:18:04.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:18:04.401+0000 7f9a1629d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 51516) UID: 1000 2024-09-12T16:18:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-12T16:18:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_23': 2024-09-12T16:18:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a52cb8369a 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a52cb8369a 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:20 2024 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:20 2024 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:20 2024 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: d6fec4d2-a64c-49a2-811d-33983c8062c0 2024-09-12T16:18:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-12T16:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.export 2024-09-12T16:18:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.export 2024-09-12T16:18:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.export 2024-09-12T16:18:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.export 2024-09-12T16:18:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.export 2024-09-12T16:18:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.export 2024-09-12T16:18:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_23.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_23.export 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_24 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-12T16:18:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-12T16:18:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-12T16:18:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-12T16:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-12T16:18:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-12T16:18:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:08.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-12T16:18:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-09-12T16:18:08.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-12T16:18:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:08.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2024-09-12T16:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_24 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_24 2024-09-12T16:18:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-12T16:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_24.status 2024-09-12T16:18:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.status 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_24.status 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-12T16:18:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 132 2024-09-12T16:18:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.status 2024-09-12T16:18:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.status 2024-09-12T16:18:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-12T16:18:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=132 2024-09-12T16:18:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-12T16:18:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=58 2024-09-12T16:18:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 132 = 58 2024-09-12T16:18:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-12T16:18:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 132 = '' 2024-09-12T16:18:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:10.502 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-12T16:18:10.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-12T16:18:10.503 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-12T16:18:10.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-12T16:18:10.506 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-12T16:18:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-12T16:18:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 132 = '' 2024-09-12T16:18:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:18:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.status 2024-09-12T16:18:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-12T16:18:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=132 2024-09-12T16:18:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 132 = 132 2024-09-12T16:18:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-12T16:18:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_24': 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5ded68f0d 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5ded68f0d 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:33 2024 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:33 2024 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:33 2024 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 31fe714a-6461-4cbe-82ca-613754e1fba5 2024-09-12T16:18:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-12T16:18:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:18:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.export 2024-09-12T16:18:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.export 2024-09-12T16:18:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.export 2024-09-12T16:18:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.export 2024-09-12T16:18:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.export 2024-09-12T16:18:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.export 2024-09-12T16:18:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_24.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_24.export 2024-09-12T16:18:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_25 2024-09-12T16:18:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-12T16:18:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-12T16:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-12T16:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:16.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-12T16:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-12T16:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-12T16:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-12T16:18:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-12T16:18:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-12T16:18:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-12T16:18:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-12T16:18:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2024-09-12T16:18:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-12T16:18:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2024-09-12T16:18:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2024-09-12T16:18:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_25 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_25 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_25.status 2024-09-12T16:18:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.status 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_25.status 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-12T16:18:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 134 2024-09-12T16:18:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.status 2024-09-12T16:18:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.status 2024-09-12T16:18:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-12T16:18:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=134 2024-09-12T16:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-12T16:18:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=59 2024-09-12T16:18:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 134 = 59 2024-09-12T16:18:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-12T16:18:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 134 = '' 2024-09-12T16:18:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-12T16:18:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 134 = '' 2024-09-12T16:18:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:18:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.status 2024-09-12T16:18:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-12T16:18:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=134 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 134 = 134 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-12T16:18:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_25': 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a511cdbd8f 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a511cdbd8f 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:10:45 2024 2024-09-12T16:18:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:10:45 2024 2024-09-12T16:18:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:10:45 2024 2024-09-12T16:18:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: e4321129-6277-4549-af49-32f7bc842699 2024-09-12T16:18:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.export 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.export 2024-09-12T16:18:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.export 2024-09-12T16:18:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.export 2024-09-12T16:18:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.export 2024-09-12T16:18:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.export 2024-09-12T16:18:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_25.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_25.export 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_26 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-12T16:18:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-12T16:18:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-12T16:18:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-12T16:18:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-12T16:18:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-12T16:18:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-12T16:18:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-12T16:18:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-12T16:18:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-12T16:18:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-12T16:18:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-09-12T16:18:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_26 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_26 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_26.status 2024-09-12T16:18:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.status 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_26.status 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.status 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-12T16:18:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 136 2024-09-12T16:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.status 2024-09-12T16:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.status 2024-09-12T16:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-12T16:18:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=136 2024-09-12T16:18:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.status 2024-09-12T16:18:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.status 2024-09-12T16:18:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-12T16:18:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 136 = '' 2024-09-12T16:18:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.status 2024-09-12T16:18:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.status 2024-09-12T16:18:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-12T16:18:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 136 = '' 2024-09-12T16:18:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.status 2024-09-12T16:18:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.status 2024-09-12T16:18:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-12T16:18:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=136 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 136 = 136 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-12T16:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_26': 2024-09-12T16:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a578ca6cf2 2024-09-12T16:18:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a578ca6cf2 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:00 2024 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:00 2024 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:00 2024 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: de6e82d5-259a-465e-adb7-ef275c6429d6 2024-09-12T16:18:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.export 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.export 2024-09-12T16:18:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.export 2024-09-12T16:18:28.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.export 2024-09-12T16:18:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.export 2024-09-12T16:18:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.export 2024-09-12T16:18:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_26.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_26.export 2024-09-12T16:18:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_27 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-12T16:18:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-12T16:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-12T16:18:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-12T16:18:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:31.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-12T16:18:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-12T16:18:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-12T16:18:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-12T16:18:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-12T16:18:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-09-12T16:18:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-09-12T16:18:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_27 2024-09-12T16:18:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_27 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-12T16:18:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_27.status 2024-09-12T16:18:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.status 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_27.status 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-12T16:18:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 138 2024-09-12T16:18:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.status 2024-09-12T16:18:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.status 2024-09-12T16:18:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-12T16:18:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=138 2024-09-12T16:18:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-12T16:18:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=61 2024-09-12T16:18:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 138 = 61 2024-09-12T16:18:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:32.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-12T16:18:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 138 = '' 2024-09-12T16:18:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-12T16:18:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 138 = '' 2024-09-12T16:18:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:18:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.status 2024-09-12T16:18:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-12T16:18:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=138 2024-09-12T16:18:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 138 = 138 2024-09-12T16:18:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-12T16:18:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-12T16:18:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_27': 2024-09-12T16:18:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a52e26b4f2 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a52e26b4f2 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:11 2024 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:11 2024 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:11 2024 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 2e3a6a3e-2072-4093-baab-81987b90abbb 2024-09-12T16:18:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.export 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.export 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.export 2024-09-12T16:18:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.export 2024-09-12T16:18:36.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.export 2024-09-12T16:18:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.export 2024-09-12T16:18:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_27.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_27.export 2024-09-12T16:18:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_28 2024-09-12T16:18:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-12T16:18:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-12T16:18:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-12T16:18:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-12T16:18:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-12T16:18:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-12T16:18:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-12T16:18:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-09-12T16:18:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-12T16:18:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2024-09-12T16:18:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2024-09-12T16:18:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_28 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_28 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-12T16:18:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_28.status 2024-09-12T16:18:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.status 2024-09-12T16:18:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-12T16:18:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_28.status 2024-09-12T16:18:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.status 2024-09-12T16:18:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-12T16:18:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-12T16:18:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 140 2024-09-12T16:18:40.482 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.481+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa59bd49c0 2024-09-12T16:18:40.485 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.484+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa59bd52b0 2024-09-12T16:18:40.485 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.484+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa59bd5860 2024-09-12T16:18:40.485 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.484+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa59bd5d40 2024-09-12T16:18:40.485 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.484+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5bb8f110 2024-09-12T16:18:40.485 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.484+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5e01edd0 2024-09-12T16:18:40.486 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.484+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa7964eb60 2024-09-12T16:18:40.486 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:18:40.484+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa8c9f9e10 2024-09-12T16:18:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.status 2024-09-12T16:18:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.status 2024-09-12T16:18:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-12T16:18:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=140 2024-09-12T16:18:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.status 2024-09-12T16:18:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.status 2024-09-12T16:18:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-12T16:18:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=62 2024-09-12T16:18:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 140 = 62 2024-09-12T16:18:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.status 2024-09-12T16:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.status 2024-09-12T16:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-12T16:18:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 140 = '' 2024-09-12T16:18:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.status 2024-09-12T16:18:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.status 2024-09-12T16:18:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-12T16:18:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=140 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 140 = 140 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-12T16:18:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_28': 2024-09-12T16:18:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5c5848d79 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5c5848d79 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:18 2024 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:18 2024 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:18 2024 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 04c258ba-62b0-4f2a-9ccc-60bd5563c0a1 2024-09-12T16:18:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.export 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.export 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.export 2024-09-12T16:18:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.export 2024-09-12T16:18:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.export 2024-09-12T16:18:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.export 2024-09-12T16:18:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_28.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_28.export 2024-09-12T16:18:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_29 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-12T16:18:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-12T16:18:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-12T16:18:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:46.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-12T16:18:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-12T16:18:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-12T16:18:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-12T16:18:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-12T16:18:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2024-09-12T16:18:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-12T16:18:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2024-09-12T16:18:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:46.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2024-09-12T16:18:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_29 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_29 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_29.status 2024-09-12T16:18:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.status 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_29.status 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.status 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-12T16:18:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 142 2024-09-12T16:18:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.status 2024-09-12T16:18:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.status 2024-09-12T16:18:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-12T16:18:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=142 2024-09-12T16:18:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.status 2024-09-12T16:18:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.status 2024-09-12T16:18:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-12T16:18:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=63 2024-09-12T16:18:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 142 = 63 2024-09-12T16:18:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.status 2024-09-12T16:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.status 2024-09-12T16:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-12T16:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 142 = '' 2024-09-12T16:18:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.status 2024-09-12T16:18:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.status 2024-09-12T16:18:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-12T16:18:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=142 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 142 = 142 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_29': 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5b2fd7593 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5b2fd7593 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:28 2024 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:28 2024 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:28 2024 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 9d01e9e8-f583-4610-9203-3eeb61dc6505 2024-09-12T16:18:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.export 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.export 2024-09-12T16:18:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.export 2024-09-12T16:18:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.export 2024-09-12T16:18:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.export 2024-09-12T16:18:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.export 2024-09-12T16:18:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_29.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_29.export 2024-09-12T16:18:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_30 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:52.498 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:18:52.496+0000 7fbc5a793640 -1 reset not still connected to 0x55f81f0fb860 2024-09-12T16:18:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-12T16:18:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-12T16:18:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:18:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:18:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:18:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:18:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:18:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:18:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:18:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:18:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:18:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:18:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:18:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-12T16:18:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-12T16:18:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:18:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:18:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:18:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:18:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:18:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-12T16:18:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-12T16:18:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:53.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:18:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:18:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:18:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-09-12T16:18:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-12T16:18:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:18:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:18:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:18:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:18:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:18:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:18:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2024-09-12T16:18:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:18:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:18:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:18:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:18:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_30 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_30 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:18:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-12T16:18:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_30.status 2024-09-12T16:18:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.status 2024-09-12T16:18:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-12T16:18:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_30.status 2024-09-12T16:18:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:18:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.status 2024-09-12T16:18:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-12T16:18:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-12T16:18:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 144 2024-09-12T16:18:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.status 2024-09-12T16:18:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.status 2024-09-12T16:18:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-12T16:18:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:18:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=144 2024-09-12T16:18:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:18:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.status 2024-09-12T16:18:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.status 2024-09-12T16:18:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-12T16:18:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:54.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:54.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 144 = '' 2024-09-12T16:18:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:18:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.status 2024-09-12T16:18:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.status 2024-09-12T16:18:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-12T16:18:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:18:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:18:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:18:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:18:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:18:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 144 = '' 2024-09-12T16:18:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:18:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:18:55.414 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3916188783435306) > weight (0.25) 2024-09-12T16:18:55.414 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8074660484455332) > weight (0.25) 2024-09-12T16:18:55.415 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6278451294419922) > weight (0.25) 2024-09-12T16:18:55.415 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-12T16:18:55.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-12T16:18:55.415 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-12T16:18:55.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:18:55.415+0000 7f041b017640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 56700) UID: 1000 2024-09-12T16:18:55.938 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.937+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b2943c70 2024-09-12T16:18:55.945 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.943+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b2943e10 2024-09-12T16:18:55.945 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.944+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b4f33c70 2024-09-12T16:18:55.947 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.946+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b5a23860 2024-09-12T16:18:55.947 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.946+0000 7f765f9cc640 -1 reset not still connected to 0x55d0bb6df930 2024-09-12T16:18:55.947 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.946+0000 7f765f9cc640 -1 reset not still connected to 0x55d0c5129ba0 2024-09-12T16:18:55.948 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.947+0000 7f765f9cc640 -1 reset not still connected to 0x55d0d8040f70 2024-09-12T16:18:55.948 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.947+0000 7f765f9cc640 -1 reset not still connected to 0x55d0d91cb110 2024-09-12T16:18:55.948 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:18:55.947+0000 7f765f9cc640 -1 reset not still connected to 0x55d0dff651e0 2024-09-12T16:18:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.status 2024-09-12T16:18:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.status 2024-09-12T16:18:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-12T16:18:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:18:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:18:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=144 2024-09-12T16:18:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 144 = 144 2024-09-12T16:18:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-12T16:18:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:18:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:18:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-12T16:18:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_30': 2024-09-12T16:18:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:18:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:18:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:18:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5cfd8564f 2024-09-12T16:18:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5cfd8564f 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:42 2024 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:42 2024 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:42 2024 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 44d6b71b-d3f9-4302-8b48-4593d4e59e2c 2024-09-12T16:18:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.export 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.export 2024-09-12T16:18:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.export 2024-09-12T16:18:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.export 2024-09-12T16:18:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.export 2024-09-12T16:18:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:18:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.export 2024-09-12T16:18:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_30.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_30.export 2024-09-12T16:18:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:18:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:18:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_31 2024-09-12T16:18:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-12T16:18:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:18:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:18:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:18:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:18:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-12T16:18:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:18:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-12T16:18:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:18:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:18:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:18:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:18:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:18:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:18:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-12T16:19:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-12T16:19:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-12T16:19:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-12T16:19:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:00.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-12T16:19:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-12T16:19:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-12T16:19:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-09-12T16:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-12T16:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2024-09-12T16:19:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_31 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_31 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-12T16:19:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_31.status 2024-09-12T16:19:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.status 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_31.status 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.status 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-12T16:19:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 146 2024-09-12T16:19:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.status 2024-09-12T16:19:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.status 2024-09-12T16:19:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-12T16:19:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=146 2024-09-12T16:19:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.status 2024-09-12T16:19:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.status 2024-09-12T16:19:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-12T16:19:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 146 = '' 2024-09-12T16:19:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.status 2024-09-12T16:19:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.status 2024-09-12T16:19:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-12T16:19:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 146 = '' 2024-09-12T16:19:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.status 2024-09-12T16:19:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.status 2024-09-12T16:19:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-12T16:19:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=146 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 146 = 146 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:03.416 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-12T16:19:03.416 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:19:03.417 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-12T16:19:03.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-12T16:19:03.417 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-12T16:19:03.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-12T16:19:03.420 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-12T16:19:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_31': 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5d699334e 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5d699334e 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:11:48 2024 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:11:48 2024 2024-09-12T16:19:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:11:48 2024 2024-09-12T16:19:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 5666a566-1e64-44f3-803d-958a51c3829a 2024-09-12T16:19:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.export 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.export 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.export 2024-09-12T16:19:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.export 2024-09-12T16:19:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.export 2024-09-12T16:19:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.export 2024-09-12T16:19:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_31.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_31.export 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_32 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-12T16:19:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-12T16:19:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-12T16:19:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-12T16:19:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-12T16:19:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-12T16:19:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-12T16:19:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-09-12T16:19:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-12T16:19:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2024-09-12T16:19:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_32 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_32 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_32.status 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.status 2024-09-12T16:19:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-12T16:19:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_32.status 2024-09-12T16:19:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.status 2024-09-12T16:19:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-12T16:19:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-12T16:19:08.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 148 2024-09-12T16:19:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.status 2024-09-12T16:19:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.status 2024-09-12T16:19:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-12T16:19:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=148 2024-09-12T16:19:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:08.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.status 2024-09-12T16:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.status 2024-09-12T16:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-12T16:19:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 148 = '' 2024-09-12T16:19:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.status 2024-09-12T16:19:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.status 2024-09-12T16:19:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-12T16:19:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 148 = '' 2024-09-12T16:19:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.status 2024-09-12T16:19:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.status 2024-09-12T16:19:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-12T16:19:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=148 2024-09-12T16:19:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 148 = 148 2024-09-12T16:19:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-12T16:19:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-12T16:19:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_32': 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a56cd5a746 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a56cd5a746 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:01 2024 2024-09-12T16:19:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:01 2024 2024-09-12T16:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:01 2024 2024-09-12T16:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 6d745883-fe82-4878-8b05-e9925084374f 2024-09-12T16:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.export 2024-09-12T16:19:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.export 2024-09-12T16:19:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.export 2024-09-12T16:19:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.export 2024-09-12T16:19:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.export 2024-09-12T16:19:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.export 2024-09-12T16:19:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_32.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_32.export 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_33 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-12T16:19:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-12T16:19:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-12T16:19:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-12T16:19:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-12T16:19:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-12T16:19:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-12T16:19:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2024-09-12T16:19:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-12T16:19:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2024-09-12T16:19:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_33 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_33 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-12T16:19:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_33.status 2024-09-12T16:19:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.status 2024-09-12T16:19:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-12T16:19:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_33.status 2024-09-12T16:19:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.status 2024-09-12T16:19:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-12T16:19:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-12T16:19:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 150 2024-09-12T16:19:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.status 2024-09-12T16:19:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.status 2024-09-12T16:19:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-12T16:19:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=150 2024-09-12T16:19:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.status 2024-09-12T16:19:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.status 2024-09-12T16:19:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-12T16:19:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 150 = '' 2024-09-12T16:19:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.status 2024-09-12T16:19:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.status 2024-09-12T16:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-12T16:19:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 150 = '' 2024-09-12T16:19:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:17.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.status 2024-09-12T16:19:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.status 2024-09-12T16:19:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-12T16:19:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=150 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 150 = 150 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-12T16:19:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_33': 2024-09-12T16:19:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a53a648fe 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a53a648fe 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:15 2024 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:15 2024 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:15 2024 2024-09-12T16:19:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 0817148d-e3ac-4c66-9784-73a94988f707 2024-09-12T16:19:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-12T16:19:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:19:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.export 2024-09-12T16:19:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.export 2024-09-12T16:19:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.export 2024-09-12T16:19:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.export 2024-09-12T16:19:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.export 2024-09-12T16:19:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.export 2024-09-12T16:19:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_33.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_33.export 2024-09-12T16:19:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_34 2024-09-12T16:19:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-12T16:19:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-12T16:19:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-12T16:19:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:20.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-12T16:19:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-12T16:19:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-12T16:19:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-12T16:19:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-12T16:19:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-12T16:19:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-09-12T16:19:21.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-12T16:19:21.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:21.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2024-09-12T16:19:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2024-09-12T16:19:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_34 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_34 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_34.status 2024-09-12T16:19:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.status 2024-09-12T16:19:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_34.status 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-12T16:19:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 152 2024-09-12T16:19:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.status 2024-09-12T16:19:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.status 2024-09-12T16:19:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-12T16:19:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=152 2024-09-12T16:19:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-12T16:19:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=68 2024-09-12T16:19:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 152 = 68 2024-09-12T16:19:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-12T16:19:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=68 2024-09-12T16:19:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 152 = 68 2024-09-12T16:19:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-12T16:19:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 152 = '' 2024-09-12T16:19:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:19:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.status 2024-09-12T16:19:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-12T16:19:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=152 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 152 = 152 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_34': 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5fd4b984b 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5fd4b984b 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:23 2024 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:23 2024 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:23 2024 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 1cffc7cb-c521-45a9-965f-d5265ebeee09 2024-09-12T16:19:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-12T16:19:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:19:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.export 2024-09-12T16:19:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.export 2024-09-12T16:19:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.export 2024-09-12T16:19:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.export 2024-09-12T16:19:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.export 2024-09-12T16:19:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.export 2024-09-12T16:19:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_34.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_34.export 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_35 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-12T16:19:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-12T16:19:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-12T16:19:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-12T16:19:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-09-12T16:19:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:29.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-09-12T16:19:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-12T16:19:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-09-12T16:19:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-12T16:19:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_35 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_35 2024-09-12T16:19:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_35 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_35 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_35.status 2024-09-12T16:19:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.status 2024-09-12T16:19:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_35.status 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-12T16:19:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 154 2024-09-12T16:19:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.status 2024-09-12T16:19:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.status 2024-09-12T16:19:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-12T16:19:30.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=154 2024-09-12T16:19:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-12T16:19:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=69 2024-09-12T16:19:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 154 = 69 2024-09-12T16:19:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-12T16:19:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 154 = '' 2024-09-12T16:19:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-12T16:19:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:31.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 154 = '' 2024-09-12T16:19:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:19:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.status 2024-09-12T16:19:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-12T16:19:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=154 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 154 = 154 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-12T16:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_35': 2024-09-12T16:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5f4655010 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5f4655010 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:29 2024 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:29 2024 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:29 2024 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 82a2b8c9-597b-4a1a-9ca6-24a0c920fc52 2024-09-12T16:19:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.export 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.export 2024-09-12T16:19:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.export 2024-09-12T16:19:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.export 2024-09-12T16:19:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.export 2024-09-12T16:19:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.export 2024-09-12T16:19:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_35.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_35.export 2024-09-12T16:19:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_36 2024-09-12T16:19:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-12T16:19:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-12T16:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-12T16:19:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-12T16:19:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-12T16:19:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-12T16:19:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-12T16:19:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-12T16:19:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-12T16:19:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-12T16:19:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-12T16:19:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-12T16:19:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_36 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_36 2024-09-12T16:19:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-12T16:19:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_36.status 2024-09-12T16:19:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.status 2024-09-12T16:19:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-12T16:19:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_36.status 2024-09-12T16:19:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-12T16:19:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-12T16:19:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 156 2024-09-12T16:19:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.status 2024-09-12T16:19:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.status 2024-09-12T16:19:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-12T16:19:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=156 2024-09-12T16:19:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-12T16:19:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=70 2024-09-12T16:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 156 = 70 2024-09-12T16:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-12T16:19:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 156 = '' 2024-09-12T16:19:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-12T16:19:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:40.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 156 = '' 2024-09-12T16:19:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:19:40.508 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T16:19:40.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T16:19:40.509 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-12T16:19:40.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:19:40.508+0000 7f1420d70640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49092) UID: 1000 2024-09-12T16:19:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.status 2024-09-12T16:19:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-12T16:19:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=156 2024-09-12T16:19:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 156 = 156 2024-09-12T16:19:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_36': 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a565db105e 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a565db105e 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:42 2024 2024-09-12T16:19:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:42 2024 2024-09-12T16:19:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:42 2024 2024-09-12T16:19:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: a0349dfc-0d36-4ede-9020-d1b706f0f9dc 2024-09-12T16:19:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.export 2024-09-12T16:19:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.export 2024-09-12T16:19:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.export 2024-09-12T16:19:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.export 2024-09-12T16:19:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.export 2024-09-12T16:19:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.export 2024-09-12T16:19:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_36.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_36.export 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_37 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-12T16:19:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-12T16:19:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-12T16:19:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-12T16:19:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-12T16:19:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-12T16:19:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-12T16:19:45.510 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T16:19:45.510 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:19:45.510 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T16:19:45.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T16:19:45.511 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T16:19:45.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T16:19:45.513 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-12T16:19:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-12T16:19:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-12T16:19:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-12T16:19:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2024-09-12T16:19:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2024-09-12T16:19:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_37 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_37 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-12T16:19:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_37.status 2024-09-12T16:19:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.status 2024-09-12T16:19:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-12T16:19:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_37.status 2024-09-12T16:19:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-12T16:19:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-12T16:19:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 158 2024-09-12T16:19:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.status 2024-09-12T16:19:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.status 2024-09-12T16:19:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-12T16:19:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=158 2024-09-12T16:19:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-12T16:19:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=71 2024-09-12T16:19:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 158 = 71 2024-09-12T16:19:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-12T16:19:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 158 = '' 2024-09-12T16:19:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-12T16:19:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:48.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 158 = '' 2024-09-12T16:19:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:48.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:19:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.status 2024-09-12T16:19:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-12T16:19:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=158 2024-09-12T16:19:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 158 = 158 2024-09-12T16:19:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-12T16:19:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-12T16:19:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_37': 2024-09-12T16:19:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a565dbd452 2024-09-12T16:19:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a565dbd452 2024-09-12T16:19:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:12:52 2024 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:12:52 2024 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:12:52 2024 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 0b18fcd6-90f2-4536-82ab-2679001c84e1 2024-09-12T16:19:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-12T16:19:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:19:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.export 2024-09-12T16:19:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.export 2024-09-12T16:19:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.export 2024-09-12T16:19:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.export 2024-09-12T16:19:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.export 2024-09-12T16:19:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.export 2024-09-12T16:19:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_37.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_37.export 2024-09-12T16:19:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_38 2024-09-12T16:19:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-12T16:19:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-12T16:19:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-12T16:19:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-12T16:19:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:19:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:19:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:19:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-12T16:19:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:19:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:19:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:19:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:19:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:19:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:19:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:19:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:19:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-12T16:19:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-12T16:19:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-12T16:19:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:19:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:19:53.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:19:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:19:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:19:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-12T16:19:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-12T16:19:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-09-12T16:19:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:19:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:19:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:19:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:19:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:19:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:19:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:19:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:19:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-09-12T16:19:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-12T16:19:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:19:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:19:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:19:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:19:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:19:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:19:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:19:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:19:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:19:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:19:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:19:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:19:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2024-09-12T16:19:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:19:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:19:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:19:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:19:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:19:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_38 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:19:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_38 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_38.status 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.status 2024-09-12T16:19:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-12T16:19:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_38.status 2024-09-12T16:19:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:19:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.status 2024-09-12T16:19:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-12T16:19:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-12T16:19:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 160 2024-09-12T16:19:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.status 2024-09-12T16:19:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.status 2024-09-12T16:19:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-12T16:19:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:19:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=160 2024-09-12T16:19:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:19:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.status 2024-09-12T16:19:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.status 2024-09-12T16:19:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-12T16:19:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=72 2024-09-12T16:19:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 160 = 72 2024-09-12T16:19:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:19:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.status 2024-09-12T16:19:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.status 2024-09-12T16:19:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-12T16:19:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:19:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:19:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:19:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:19:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:19:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 160 = '' 2024-09-12T16:19:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:19:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:19:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.status 2024-09-12T16:19:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.status 2024-09-12T16:19:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-12T16:19:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:19:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:19:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=160 2024-09-12T16:19:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 160 = 160 2024-09-12T16:19:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:19:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:19:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-12T16:19:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_38': 2024-09-12T16:19:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:19:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a59aaf99ac 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a59aaf99ac 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:02 2024 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:02 2024 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:02 2024 2024-09-12T16:19:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:19:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:19:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:19:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: e0e6fdb8-ef78-418e-9cc0-4afd4bd69e5b 2024-09-12T16:19:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:19:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.export 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.export 2024-09-12T16:19:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.export 2024-09-12T16:19:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.export 2024-09-12T16:19:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.export 2024-09-12T16:19:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:19:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.export 2024-09-12T16:19:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_38.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_38.export 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_39 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:19:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:19:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:19:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:19:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-12T16:20:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-12T16:20:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-12T16:20:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:00.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-12T16:20:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-12T16:20:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-12T16:20:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-12T16:20:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-12T16:20:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_39 2024-09-12T16:20:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_39 2024-09-12T16:20:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_39 2024-09-12T16:20:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:00.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_39 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_39.status 2024-09-12T16:20:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.status 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_39.status 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.status 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-12T16:20:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 162 2024-09-12T16:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.status 2024-09-12T16:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.status 2024-09-12T16:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-12T16:20:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=162 2024-09-12T16:20:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.status 2024-09-12T16:20:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.status 2024-09-12T16:20:01.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-12T16:20:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=73 2024-09-12T16:20:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 162 = 73 2024-09-12T16:20:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.status 2024-09-12T16:20:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.status 2024-09-12T16:20:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-12T16:20:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 162 = '' 2024-09-12T16:20:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.status 2024-09-12T16:20:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.status 2024-09-12T16:20:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-12T16:20:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=162 2024-09-12T16:20:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 162 = 162 2024-09-12T16:20:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-12T16:20:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_39': 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a59405d4db 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a59405d4db 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:14 2024 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:14 2024 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:14 2024 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: d0fe3ef0-64f0-4fc2-be1f-5b61839e3452 2024-09-12T16:20:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.export 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.export 2024-09-12T16:20:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.export 2024-09-12T16:20:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.export 2024-09-12T16:20:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.export 2024-09-12T16:20:04.421 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5945150869919581) > weight (0.25) 2024-09-12T16:20:04.422 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-12T16:20:04.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-12T16:20:04.422 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-12T16:20:04.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:20:04.422+0000 7f43e555d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 50399) UID: 1000 2024-09-12T16:20:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.export 2024-09-12T16:20:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_39.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_39.export 2024-09-12T16:20:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_40 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:05.423 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-12T16:20:05.423 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:20:05.423 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-12T16:20:05.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-12T16:20:05.423 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-12T16:20:05.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-12T16:20:05.426 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-12T16:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-12T16:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-12T16:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-12T16:20:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-12T16:20:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-12T16:20:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-12T16:20:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-12T16:20:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-12T16:20:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-12T16:20:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-12T16:20:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2024-09-12T16:20:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_40 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_40 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-12T16:20:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_40.status 2024-09-12T16:20:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.status 2024-09-12T16:20:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-12T16:20:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_40.status 2024-09-12T16:20:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.status 2024-09-12T16:20:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-12T16:20:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-12T16:20:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 164 2024-09-12T16:20:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.status 2024-09-12T16:20:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.status 2024-09-12T16:20:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-12T16:20:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=164 2024-09-12T16:20:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.status 2024-09-12T16:20:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.status 2024-09-12T16:20:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-12T16:20:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 164 = '' 2024-09-12T16:20:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.status 2024-09-12T16:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.status 2024-09-12T16:20:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-12T16:20:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 164 = '' 2024-09-12T16:20:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.status 2024-09-12T16:20:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.status 2024-09-12T16:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-12T16:20:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=164 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 164 = 164 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-12T16:20:11.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_40': 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5c5dc2a1a 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5c5dc2a1a 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:20 2024 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:20 2024 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:20 2024 2024-09-12T16:20:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: b72eb11e-35a3-4497-848c-fa02d50ba2c9 2024-09-12T16:20:11.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.export 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.export 2024-09-12T16:20:11.344 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.export 2024-09-12T16:20:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.export 2024-09-12T16:20:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.export 2024-09-12T16:20:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.export 2024-09-12T16:20:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_40.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_40.export 2024-09-12T16:20:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_41 2024-09-12T16:20:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-12T16:20:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-12T16:20:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-12T16:20:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-12T16:20:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-12T16:20:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-12T16:20:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-12T16:20:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-12T16:20:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-12T16:20:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-12T16:20:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-12T16:20:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_41 2024-09-12T16:20:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_41 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-12T16:20:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_41.status 2024-09-12T16:20:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.status 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_41.status 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.status 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-12T16:20:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 166 2024-09-12T16:20:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.status 2024-09-12T16:20:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.status 2024-09-12T16:20:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-12T16:20:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=166 2024-09-12T16:20:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.status 2024-09-12T16:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.status 2024-09-12T16:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-12T16:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 166 = '' 2024-09-12T16:20:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.status 2024-09-12T16:20:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:16.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.status 2024-09-12T16:20:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-12T16:20:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 166 = '' 2024-09-12T16:20:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.status 2024-09-12T16:20:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.status 2024-09-12T16:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-12T16:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=166 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 166 = 166 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-12T16:20:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_41': 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5af2b7d92 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5af2b7d92 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:31 2024 2024-09-12T16:20:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:31 2024 2024-09-12T16:20:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:31 2024 2024-09-12T16:20:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 208f0c1a-c137-477d-8165-0c0b6213b01f 2024-09-12T16:20:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.export 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.export 2024-09-12T16:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.export 2024-09-12T16:20:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.export 2024-09-12T16:20:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.export 2024-09-12T16:20:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.export 2024-09-12T16:20:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_41.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_41.export 2024-09-12T16:20:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_42 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-12T16:20:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-12T16:20:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-12T16:20:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-12T16:20:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-12T16:20:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-12T16:20:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-12T16:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:21.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:21.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-12T16:20:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-12T16:20:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_42 2024-09-12T16:20:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_42 2024-09-12T16:20:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_42 2024-09-12T16:20:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_42 2024-09-12T16:20:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-12T16:20:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_42.status 2024-09-12T16:20:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.status 2024-09-12T16:20:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-12T16:20:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_42.status 2024-09-12T16:20:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.status 2024-09-12T16:20:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-12T16:20:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-12T16:20:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 168 2024-09-12T16:20:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.status 2024-09-12T16:20:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.status 2024-09-12T16:20:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-12T16:20:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=168 2024-09-12T16:20:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.status 2024-09-12T16:20:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.status 2024-09-12T16:20:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-12T16:20:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 168 = '' 2024-09-12T16:20:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.status 2024-09-12T16:20:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.status 2024-09-12T16:20:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-12T16:20:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 168 = '' 2024-09-12T16:20:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.status 2024-09-12T16:20:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.status 2024-09-12T16:20:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-12T16:20:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=168 2024-09-12T16:20:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 168 = 168 2024-09-12T16:20:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-12T16:20:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:24.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-12T16:20:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_42': 2024-09-12T16:20:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a56f64ead2 2024-09-12T16:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a56f64ead2 2024-09-12T16:20:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:25.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:44 2024 2024-09-12T16:20:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:44 2024 2024-09-12T16:20:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:44 2024 2024-09-12T16:20:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 13c21475-07e1-46bd-818c-3e23de114deb 2024-09-12T16:20:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-12T16:20:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:20:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.export 2024-09-12T16:20:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.export 2024-09-12T16:20:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.export 2024-09-12T16:20:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.export 2024-09-12T16:20:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.export 2024-09-12T16:20:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.export 2024-09-12T16:20:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_42.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_42.export 2024-09-12T16:20:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_43 2024-09-12T16:20:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-12T16:20:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-12T16:20:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-12T16:20:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-12T16:20:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-12T16:20:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:28.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-12T16:20:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:29.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-12T16:20:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-12T16:20:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-12T16:20:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:29.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-12T16:20:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-12T16:20:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-09-12T16:20:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:30.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-09-12T16:20:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-12T16:20:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_43 2024-09-12T16:20:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_43 2024-09-12T16:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_43 2024-09-12T16:20:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:30.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_43 2024-09-12T16:20:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-12T16:20:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_43.status 2024-09-12T16:20:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.status 2024-09-12T16:20:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-12T16:20:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_43.status 2024-09-12T16:20:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.status 2024-09-12T16:20:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-12T16:20:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-12T16:20:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 170 2024-09-12T16:20:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.status 2024-09-12T16:20:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.status 2024-09-12T16:20:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-12T16:20:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=170 2024-09-12T16:20:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.status 2024-09-12T16:20:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.status 2024-09-12T16:20:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-12T16:20:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 170 = '' 2024-09-12T16:20:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.status 2024-09-12T16:20:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.status 2024-09-12T16:20:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-12T16:20:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 170 = '' 2024-09-12T16:20:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.status 2024-09-12T16:20:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.status 2024-09-12T16:20:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-12T16:20:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=170 2024-09-12T16:20:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 170 = 170 2024-09-12T16:20:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-12T16:20:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-12T16:20:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_43': 2024-09-12T16:20:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5e7b86bb6 2024-09-12T16:20:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5e7b86bb6 2024-09-12T16:20:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:13:50 2024 2024-09-12T16:20:32.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:13:50 2024 2024-09-12T16:20:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:13:50 2024 2024-09-12T16:20:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 5e72a01d-2edd-4c16-ac2b-2b4399cf2dd2 2024-09-12T16:20:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-12T16:20:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:20:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.export 2024-09-12T16:20:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.export 2024-09-12T16:20:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.export 2024-09-12T16:20:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.export 2024-09-12T16:20:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.export 2024-09-12T16:20:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.export 2024-09-12T16:20:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_43.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_43.export 2024-09-12T16:20:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_44 2024-09-12T16:20:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-12T16:20:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:34.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-12T16:20:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-12T16:20:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-12T16:20:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-12T16:20:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-12T16:20:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-12T16:20:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-12T16:20:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-12T16:20:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-12T16:20:36.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-12T16:20:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:36.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-09-12T16:20:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-09-12T16:20:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-12T16:20:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2024-09-12T16:20:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:37.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2024-09-12T16:20:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_44 2024-09-12T16:20:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_44 2024-09-12T16:20:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-12T16:20:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_44.status 2024-09-12T16:20:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.status 2024-09-12T16:20:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-12T16:20:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_44.status 2024-09-12T16:20:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.status 2024-09-12T16:20:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-12T16:20:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:38.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-12T16:20:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 172 2024-09-12T16:20:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.status 2024-09-12T16:20:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.status 2024-09-12T16:20:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-12T16:20:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=172 2024-09-12T16:20:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.status 2024-09-12T16:20:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.status 2024-09-12T16:20:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-12T16:20:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 172 = '' 2024-09-12T16:20:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:38.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.status 2024-09-12T16:20:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.status 2024-09-12T16:20:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-12T16:20:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 172 = '' 2024-09-12T16:20:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.status 2024-09-12T16:20:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.status 2024-09-12T16:20:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-12T16:20:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=172 2024-09-12T16:20:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 172 = 172 2024-09-12T16:20:39.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-12T16:20:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:39.167 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.838+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5059652b0 2024-09-12T16:20:39.174 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.838+0000 7fae4b5c9640 -1 reset not still connected to 0x55c505965450 2024-09-12T16:20:39.181 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.838+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5059655f0 2024-09-12T16:20:39.189 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.838+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5060968f0 2024-09-12T16:20:39.197 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.839+0000 7fae4b5c9640 -1 reset not still connected to 0x55c506096b60 2024-09-12T16:20:39.204 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.839+0000 7fae4b5c9640 -1 reset not still connected to 0x55c506096f70 2024-09-12T16:20:39.212 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.839+0000 7fae4b5c9640 -1 reset not still connected to 0x55c506097110 2024-09-12T16:20:39.219 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.840+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5060971e0 2024-09-12T16:20:39.227 INFO:tasks.ceph.cluster1.osd.0.smithi016.stderr:2024-09-12T16:20:38.840+0000 7fae4b5c9640 -1 reset not still connected to 0x55c5067f1110 2024-09-12T16:20:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-12T16:20:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_44': 2024-09-12T16:20:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a56deeae3b 2024-09-12T16:20:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a56deeae3b 2024-09-12T16:20:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:03 2024 2024-09-12T16:20:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:03 2024 2024-09-12T16:20:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:03 2024 2024-09-12T16:20:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:39.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 744e74ce-448f-4d0c-b2d1-b1a3be444e01 2024-09-12T16:20:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-12T16:20:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:20:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.export 2024-09-12T16:20:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.export 2024-09-12T16:20:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.export 2024-09-12T16:20:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.export 2024-09-12T16:20:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.export 2024-09-12T16:20:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.export 2024-09-12T16:20:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_44.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_44.export 2024-09-12T16:20:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_45 2024-09-12T16:20:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-12T16:20:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-12T16:20:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:41.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-12T16:20:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-12T16:20:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-12T16:20:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-12T16:20:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-12T16:20:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-12T16:20:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-12T16:20:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:43.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-12T16:20:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-12T16:20:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-12T16:20:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:44.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-12T16:20:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-12T16:20:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2024-09-12T16:20:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:44.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:44.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:44.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2024-09-12T16:20:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_45 2024-09-12T16:20:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_45 2024-09-12T16:20:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-12T16:20:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_45.status 2024-09-12T16:20:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:45.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.status 2024-09-12T16:20:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-12T16:20:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_45.status 2024-09-12T16:20:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-12T16:20:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-12T16:20:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 174 2024-09-12T16:20:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.status 2024-09-12T16:20:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.status 2024-09-12T16:20:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-12T16:20:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=174 2024-09-12T16:20:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-12T16:20:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=79 2024-09-12T16:20:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 174 = 79 2024-09-12T16:20:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-12T16:20:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 174 = '' 2024-09-12T16:20:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-12T16:20:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 174 = '' 2024-09-12T16:20:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:20:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.status 2024-09-12T16:20:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-12T16:20:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=174 2024-09-12T16:20:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 174 = 174 2024-09-12T16:20:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-12T16:20:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-12T16:20:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_45': 2024-09-12T16:20:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a5c3af8c94 2024-09-12T16:20:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a5c3af8c94 2024-09-12T16:20:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:12 2024 2024-09-12T16:20:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:12 2024 2024-09-12T16:20:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:12 2024 2024-09-12T16:20:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:47.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: ad26f91c-a3fe-4c1d-910b-f56df2175de9 2024-09-12T16:20:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-12T16:20:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:20:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.export 2024-09-12T16:20:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.export 2024-09-12T16:20:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.export 2024-09-12T16:20:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.export 2024-09-12T16:20:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.export 2024-09-12T16:20:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.export 2024-09-12T16:20:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_45.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_45.export 2024-09-12T16:20:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_46 2024-09-12T16:20:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-12T16:20:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-12T16:20:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-12T16:20:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-12T16:20:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:51.524 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8782990395715444) > weight (0.25) 2024-09-12T16:20:51.530 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5481874535632502) > weight (0.25) 2024-09-12T16:20:51.537 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-12T16:20:51.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-12T16:20:51.552 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-12T16:20:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-12T16:20:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-12T16:20:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:51.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:20:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:20:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:20:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:20:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-12T16:20:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-12T16:20:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-12T16:20:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:20:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:20:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:20:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:20:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:20:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:20:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:20:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:20:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:20:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:20:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:20:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:20:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:20:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:20:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:20:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-12T16:20:52.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:20:51.544+0000 7f83a19d6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 53678) UID: 1000 2024-09-12T16:20:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-12T16:20:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:20:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:20:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-12T16:20:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:20:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:20:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:20:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:20:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:20:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:20:53.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:20:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:20:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:20:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:20:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:20:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:20:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:20:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:20:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-12T16:20:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-12T16:20:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-09-12T16:20:53.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:20:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:53.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:20:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:20:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:20:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:20:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:20:53.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:20:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:20:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:20:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:20:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:20:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:20:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:20:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:20:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:20:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:20:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:20:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-09-12T16:20:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:20:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:20:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:20:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:20:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:20:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:20:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:20:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_46 2024-09-12T16:20:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:20:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:20:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_46 2024-09-12T16:20:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:20:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:20:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:20:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-12T16:20:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_46.status 2024-09-12T16:20:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.status 2024-09-12T16:20:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-12T16:20:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_46.status 2024-09-12T16:20:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:20:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-12T16:20:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-12T16:20:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 176 2024-09-12T16:20:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.status 2024-09-12T16:20:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.status 2024-09-12T16:20:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-12T16:20:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:20:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=176 2024-09-12T16:20:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:20:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-12T16:20:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=80 2024-09-12T16:20:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 176 = 80 2024-09-12T16:20:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:20:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-12T16:20:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 176 = '' 2024-09-12T16:20:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:20:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-12T16:20:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:20:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:20:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:20:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:20:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:20:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 176 = '' 2024-09-12T16:20:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:20:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:20:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.status 2024-09-12T16:20:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-12T16:20:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:20:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:20:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=176 2024-09-12T16:20:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 176 = 176 2024-09-12T16:20:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-12T16:20:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:20:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:56.560 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-12T16:20:56.567 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:20:56.579 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-12T16:20:56.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-12T16:20:56.670 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-12T16:20:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-12T16:20:56.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-12T16:20:56.700 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-12T16:20:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_46': 2024-09-12T16:20:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:20:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:20:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:20:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a56aae3ca5 2024-09-12T16:20:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a56aae3ca5 2024-09-12T16:20:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:20:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:20:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:20:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:20:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:23 2024 2024-09-12T16:20:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:23 2024 2024-09-12T16:20:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:23 2024 2024-09-12T16:20:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:20:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:20:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:20:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: c85a9f05-e741-40a3-80f4-86c92eb59802 2024-09-12T16:20:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:20:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-12T16:20:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:20:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:20:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.export 2024-09-12T16:20:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.export 2024-09-12T16:20:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.export 2024-09-12T16:20:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.export 2024-09-12T16:20:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.export 2024-09-12T16:20:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:20:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.export 2024-09-12T16:20:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_46.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_46.export 2024-09-12T16:20:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:20:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:20:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_47 2024-09-12T16:20:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-12T16:20:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:20:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:20:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:20:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-12T16:20:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:20:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-12T16:20:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:20:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:20:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:20:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:20:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-12T16:20:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:20:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:20:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:20:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:20:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:20:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:20:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-12T16:20:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:20:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:20:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:20:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:20:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:20:59.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:20:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:20:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:20:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:20:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:20:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-12T16:20:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:20:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:20:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:20:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:20:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:20:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:20:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:20:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:20:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:20:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:20:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:20:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:20:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:20:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:20:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:20:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:20:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:20:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:20:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:20:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:20:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:20:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:20:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:00.003 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:21:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:21:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:21:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:21:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:21:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:00.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:00.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-12T16:21:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-12T16:21:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:00.218 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-12T16:21:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:21:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:21:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:21:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:21:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:21:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:21:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:21:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:21:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:21:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:21:00.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:21:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:21:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-12T16:21:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-12T16:21:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:00.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-12T16:21:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:21:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:21:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:21:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:21:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:21:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:21:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:21:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:21:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:21:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:01.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:21:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:21:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:21:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-12T16:21:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-12T16:21:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-12T16:21:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:21:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:21:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:21:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:21:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:21:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:21:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:21:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:21:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:21:01.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:21:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:21:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:21:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:21:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:21:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_47 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_47 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_47.status 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.status 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_47.status 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-12T16:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 178 2024-09-12T16:21:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.status 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.status 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=178 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=81 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 178 = 81 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 178 = '' 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:21:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-12T16:21:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:21:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:21:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:21:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:21:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 178 = '' 2024-09-12T16:21:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:02.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:21:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.status 2024-09-12T16:21:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-12T16:21:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=178 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 178 = 178 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:21:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:21:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:21:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_47': 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a59ec7ff1d 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a59ec7ff1d 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:21:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:29 2024 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:29 2024 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:29 2024 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: eac71c3a-77dc-48b2-a8e3-c7000c1abbd9 2024-09-12T16:21:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.export 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.export 2024-09-12T16:21:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.export 2024-09-12T16:21:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.export 2024-09-12T16:21:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.export 2024-09-12T16:21:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.export 2024-09-12T16:21:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_47.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_47.export 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_48 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:21:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-12T16:21:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-12T16:21:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:21:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:21:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:21:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:21:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-12T16:21:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:21:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:21:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:21:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-12T16:21:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:21:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:21:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:21:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:21:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:21:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:21:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:21:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:08.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:21:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:21:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-12T16:21:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-12T16:21:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:21:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:21:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:21:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-12T16:21:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-12T16:21:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:21:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:21:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-09-12T16:21:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-12T16:21:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:21:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:21:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:21:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2024-09-12T16:21:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:21:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:21:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:21:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:21:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:21:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_48 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_48 2024-09-12T16:21:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-12T16:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_48.status 2024-09-12T16:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.status 2024-09-12T16:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-12T16:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_48.status 2024-09-12T16:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.status 2024-09-12T16:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-12T16:21:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-12T16:21:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 180 2024-09-12T16:21:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.status 2024-09-12T16:21:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.status 2024-09-12T16:21:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-12T16:21:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:21:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=180 2024-09-12T16:21:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.status 2024-09-12T16:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.status 2024-09-12T16:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-12T16:21:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=82 2024-09-12T16:21:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 180 = 82 2024-09-12T16:21:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:09.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:21:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.status 2024-09-12T16:21:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.status 2024-09-12T16:21:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-12T16:21:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 180 = '' 2024-09-12T16:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:21:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.status 2024-09-12T16:21:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.status 2024-09-12T16:21:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-12T16:21:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=180 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 180 = 180 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:21:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:21:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:21:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_48': 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a52707de75 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a52707de75 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:21:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:36 2024 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:36 2024 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:36 2024 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 9815c7d5-2e9a-44c3-b4bc-6316340f81e4 2024-09-12T16:21:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:21:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-12T16:21:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:21:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:21:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.export 2024-09-12T16:21:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.export 2024-09-12T16:21:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.export 2024-09-12T16:21:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.export 2024-09-12T16:21:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.export 2024-09-12T16:21:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.export 2024-09-12T16:21:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_48.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_48.export 2024-09-12T16:21:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_49 2024-09-12T16:21:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-12T16:21:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:21:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-12T16:21:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:21:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:21:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-12T16:21:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:21:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:21:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:21:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-12T16:21:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:21:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:21:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:21:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:21:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:21:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:21:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:21:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:21:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:21:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:21:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:21:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:21:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:21:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-12T16:21:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-12T16:21:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-12T16:21:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:21:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:21:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:21:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:21:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-12T16:21:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-12T16:21:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:21:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-09-12T16:21:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-12T16:21:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:21:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:21:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:21:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2024-09-12T16:21:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:21:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:21:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:21:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:21:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:21:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:21:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:21:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_49 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_49 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-12T16:21:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_49.status 2024-09-12T16:21:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.status 2024-09-12T16:21:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-12T16:21:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_49.status 2024-09-12T16:21:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-12T16:21:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-12T16:21:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 182 2024-09-12T16:21:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.status 2024-09-12T16:21:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.status 2024-09-12T16:21:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-12T16:21:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:21:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=182 2024-09-12T16:21:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:21:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-12T16:21:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=83 2024-09-12T16:21:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 182 = 83 2024-09-12T16:21:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:21:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-12T16:21:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:21:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:21:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:21:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:21:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 182 = '' 2024-09-12T16:21:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:21:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-12T16:21:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:21:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:21:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:21:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 182 = '' 2024-09-12T16:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1.6 2024-09-12T16:21:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.status 2024-09-12T16:21:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-12T16:21:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=182 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 182 = 182 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_49': 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a8b3894510 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a8b3894510 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:21:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:41 2024 2024-09-12T16:21:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:41 2024 2024-09-12T16:21:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:41 2024 2024-09-12T16:21:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:21:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:21:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:21:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: cb8151fa-d57f-4d3c-8f0f-b2e94e60e773 2024-09-12T16:21:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:21:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-12T16:21:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:21:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:21:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.export 2024-09-12T16:21:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.export 2024-09-12T16:21:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.export 2024-09-12T16:21:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.export 2024-09-12T16:21:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.export 2024-09-12T16:21:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.export 2024-09-12T16:21:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_49.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_49.export 2024-09-12T16:21:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_50 2024-09-12T16:21:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-12T16:21:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-12T16:21:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-12T16:21:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=started 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-12T16:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=started 2024-09-12T16:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_result 2024-09-12T16:21:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=stopped 2024-09-12T16:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-12T16:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -i state 2024-09-12T16:21:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_instance=cluster1 2024-09-12T16:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance=cluster1 2024-09-12T16:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local loop_instance 2024-09-12T16:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-12T16:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sleep 0 2024-09-12T16:21:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-12T16:21:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ seq 0 3 2024-09-12T16:21:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-12T16:21:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:0 2024-09-12T16:21:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:0 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=0 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:0 2024-09-12T16:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:0 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=0 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=0 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=0 2024-09-12T16:21:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-12T16:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-12T16:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-12T16:21:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-12T16:21:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:1 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:1 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:1 2024-09-12T16:21:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=1 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=1 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=1 2024-09-12T16:21:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-12T16:21:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-12T16:21:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-12T16:21:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-12T16:21:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:2 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:2 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=2 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=2 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:2 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:2 2024-09-12T16:21:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=2 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=2 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=2 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-12T16:21:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-12T16:21:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-12T16:21:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd mirror status : get status for rbd mirror 2024-09-12T16:21:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:admin_socket: invalid command 2024-09-12T16:21:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:21:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-09-12T16:21:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1:3 2024-09-12T16:21:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ shift 2024-09-12T16:21:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance 2024-09-12T16:21:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local val=cluster1:3 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster_var_name=cluster 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local instance_var_name=instance 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=3 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval cluster=cluster1 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ eval instance=3 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-12T16:21:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local local_cluster=cluster1:3 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster=cluster1 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local val=cluster1:3 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local cluster_var_name=local_cluster 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ local instance_var_name=instance 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ cluster=cluster1 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ instance=3 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval local_cluster=cluster1 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ local_cluster=cluster1 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ eval instance=3 2024-09-12T16:21:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ instance=3 2024-09-12T16:21:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-12T16:21:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-12T16:21:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-09-12T16:21:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ return 0 2024-09-12T16:21:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ status_result=' "state": "Replaying"' 2024-09-12T16:21:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo ' "state": "Replaying"' 2024-09-12T16:21:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ grep -i Replaying 2024-09-12T16:21:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: "state": "Replaying" 2024-09-12T16:21:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=started 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test started = started 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_50 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = journal ']' 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' snapshot = snapshot ']' 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_50 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_cluster=cluster1 2024-09-12T16:21:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_pool=mirror 2024-09-12T16:21:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local remote_pool=mirror 2024-09-12T16:21:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-12T16:21:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster2-mirror-image_50.status 2024-09-12T16:21:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local status_log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.status 2024-09-12T16:21:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-12T16:21:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ echo cluster1-mirror-image_50.status 2024-09-12T16:21:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-12T16:21:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local local_status_log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.status 2024-09-12T16:21:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-12T16:21:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-12T16:21:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Snapshot ID: 184 2024-09-12T16:21:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.status 2024-09-12T16:21:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.status 2024-09-12T16:21:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-12T16:21:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-12T16:21:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snapshot_id=184 2024-09-12T16:21:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.2 2024-09-12T16:21:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.status 2024-09-12T16:21:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.status 2024-09-12T16:21:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-12T16:21:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:21:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:21:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:21:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:21:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 184 = '' 2024-09-12T16:21:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.4 2024-09-12T16:21:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.status 2024-09-12T16:21:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.status 2024-09-12T16:21:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-12T16:21:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:21:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:-:1.1: Document is empty 2024-09-12T16:21:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: 2024-09-12T16:21:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:^ 2024-09-12T16:21:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id= 2024-09-12T16:21:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 184 = '' 2024-09-12T16:21:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-12T16:21:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.8 2024-09-12T16:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.status 2024-09-12T16:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local log=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.status 2024-09-12T16:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-12T16:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-12T16:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local primary_snapshot_id=184 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 184 = 184 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-12T16:21:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:21:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-12T16:21:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_50': 2024-09-12T16:21:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 128 MiB in 32 objects 2024-09-12T16:21:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:21:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 2 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 10a89a5e9ee7 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.10a89a5e9ee7 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:14:52 2024 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:14:52 2024 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:14:52 2024 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: protected: False 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 7e4292a4-067e-49ff-9c07-bb55509086c8 2024-09-12T16:21:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: false 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap_name=snap 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local ret=0 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local rmt_export=/tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.export 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local loc_export=/tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.export 2024-09-12T16:21:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.export 2024-09-12T16:21:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.export 2024-09-12T16:21:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.export 2024-09-12T16:21:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-12T16:21:27.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cmp /tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.export 2024-09-12T16:21:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -f /tmp/tmp.65UL20DnMX/cluster2-mirror-image_50.export /tmp/tmp.65UL20DnMX/cluster1-mirror-image_50.export 2024-09-12T16:21:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ testlog 'TEST: delete many images' 2024-09-12T16:21:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ tee -a /tmp/tmp.65UL20DnMX/rbd-mirror.test.log 2024-09-12T16:21:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ date '+%F %T' 2024-09-12T16:21:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo 2024-09-12 16:21:27 TEST: delete many images 2024-09-12T16:21:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:2024-09-12 16:21:27 TEST: delete many images 2024-09-12T16:21:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 50 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_1 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-12T16:21:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:28.470 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:21:28.467+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa59bd49c0 2024-09-12T16:21:28.475 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:21:28.470+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa59bd5860 2024-09-12T16:21:28.475 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:21:28.470+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa59bd5d40 2024-09-12T16:21:28.475 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:21:28.470+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5bb8f110 2024-09-12T16:21:28.475 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:21:28.470+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5e01edd0 2024-09-12T16:21:28.475 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:21:28.470+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa7964eb60 2024-09-12T16:21:28.475 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:21:28.470+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa8c9f9e10 2024-09-12T16:21:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-12T16:21:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:21:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_1 2024-09-12T16:21:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:21:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-12T16:21:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-12T16:21:28.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:21:28.626+0000 7f40b94a4640 -1 rbd::mirror::ImageReplayer: 0x5629b1edaf00 [3/135c0a36-496a-4b49-bf12-e55a9a725020] start: already running 2024-09-12T16:21:28.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:21:28.626+0000 7f40b94a4640 -1 rbd::mirror::ImageReplayer: 0x5629b1edaf00 [3/135c0a36-496a-4b49-bf12-e55a9a725020] start: already running 2024-09-12T16:21:28.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:28.658+0000 7f97b63b9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b481380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:28.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:28.658+0000 7f97bcbc6640 -1 rbd::mirror::ImageReplayer: 0x562f4b738780 [3/135c0a36-496a-4b49-bf12-e55a9a725020] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_2 2024-09-12T16:21:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-12T16:21:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:21:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-12T16:21:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-12T16:21:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:21:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_2 2024-09-12T16:21:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:21:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-12T16:21:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-12T16:21:31.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:31.666+0000 7f97b5bb8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b494ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:31.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:31.666+0000 7f97bcbc6640 -1 rbd::mirror::ImageReplayer: 0x562f4b903180 [3/d49cf199-0fac-4fe0-afa3-c988aa8578d1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_3 2024-09-12T16:21:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-12T16:21:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:21:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-12T16:21:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-12T16:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_3 2024-09-12T16:21:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:21:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-12T16:21:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-12T16:21:34.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:34.895+0000 7f97b5bb8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b495040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:34.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:34.895+0000 7f97bcbc6640 -1 rbd::mirror::ImageReplayer: 0x562f4b739400 [3/50c9ce9d-ad98-4a52-8237-144bfb366358] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_4 2024-09-12T16:21:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-12T16:21:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:21:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-12T16:21:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-12T16:21:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:21:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_4 2024-09-12T16:21:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:21:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-12T16:21:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-12T16:21:37.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:37.691+0000 7f97b63b9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b3c41a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:37.692 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:37.691+0000 7f97bc3c5640 -1 rbd::mirror::ImageReplayer: 0x562f4b738c80 [3/2f93870e-9268-4e8c-9b38-0d6cd6a7dd81] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_5 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-12T16:21:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-12T16:21:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:21:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_5 2024-09-12T16:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-12T16:21:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-12T16:21:40.701 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:40.700+0000 7f97b5bb8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b12e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:40.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:40.700+0000 7f97bcbc6640 -1 rbd::mirror::ImageReplayer: 0x562f4b739180 [3/4e25a06d-cee5-4e43-8e00-5e4bc8c2f14a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_6 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-12T16:21:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-12T16:21:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:21:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_6 2024-09-12T16:21:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:21:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-12T16:21:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-12T16:21:43.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:43.716+0000 7f97b63b9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b4941a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:43.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:43.716+0000 7f97bcbc6640 -1 rbd::mirror::ImageReplayer: 0x562f4b739680 [3/7116ac0d-b88f-466e-8403-60e8e5d9a276] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_7 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-12T16:21:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-12T16:21:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:21:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_7 2024-09-12T16:21:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:21:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-12T16:21:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-12T16:21:46.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:46.720+0000 7f97b5bb8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b494340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:46.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:46.720+0000 7f97bc3c5640 -1 rbd::mirror::ImageReplayer: 0x562f4b739b80 [3/9ed94e20-8486-411e-a66b-35ec4ef5bf45] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_8 2024-09-12T16:21:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-12T16:21:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:21:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-12T16:21:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-12T16:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_8 2024-09-12T16:21:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:21:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-12T16:21:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-12T16:21:49.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:49.685+0000 7f97b5bb8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b3c4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:49.686 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:49.685+0000 7f97bc3c5640 -1 rbd::mirror::ImageReplayer: 0x562f4b739900 [3/7246287d-3dc0-422d-9d93-915c9d37781e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_9 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-12T16:21:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-12T16:21:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:21:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_9 2024-09-12T16:21:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:21:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-12T16:21:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-12T16:21:52.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:52.756+0000 7f97b5bb8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b4944e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:52.757 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:52.756+0000 7f97bc3c5640 -1 rbd::mirror::ImageReplayer: 0x562f4b902280 [3/f574749e-1611-4718-9f27-0329fe616606] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_10 2024-09-12T16:21:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-12T16:21:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:21:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-12T16:21:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-12T16:21:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:21:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_10 2024-09-12T16:21:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:21:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-12T16:21:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-12T16:21:55.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:55.768+0000 7f97b63b9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b3c4000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:55.769 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:55.768+0000 7f97bc3c5640 -1 rbd::mirror::ImageReplayer: 0x562f4b902000 [3/c558b2dc-a46e-4a16-bc7f-98af17a10d37] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:21:57.583 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:21:57.581+0000 7f765f9cc640 -1 reset not still connected to 0x55d0b5a23a00 2024-09-12T16:21:57.583 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:21:57.581+0000 7f765f9cc640 -1 reset not still connected to 0x55d0bb6dfad0 2024-09-12T16:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_11 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:21:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-12T16:21:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-12T16:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:21:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_11 2024-09-12T16:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:21:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-12T16:21:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-12T16:21:58.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:21:58.751+0000 7f40b8ca3640 -1 rbd::mirror::ImageReplayer: 0x5629b207d400 [3/c7fac33d-e73d-4d4b-8daf-469b5cf5e4e4] start: already running 2024-09-12T16:21:58.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:58.784+0000 7f97b63b9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b2eab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:21:58.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:21:58.784+0000 7f97bcbc6640 -1 rbd::mirror::ImageReplayer: 0x562f4b902c80 [3/c7fac33d-e73d-4d4b-8daf-469b5cf5e4e4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_12 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-12T16:22:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-12T16:22:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:22:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_12 2024-09-12T16:22:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:22:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-12T16:22:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-12T16:22:01.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:22:01.693+0000 7f97b63b9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562f4b494680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:01.694 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:22:01.693+0000 7f97bcbc6640 -1 rbd::mirror::ImageReplayer: 0x562f4b902f00 [3/c928224c-98a5-45be-8f56-7a1ae44aa427] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:03.427 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.37265416928590445) > weight (0.25) 2024-09-12T16:22:03.427 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.45482655314005427) > weight (0.25) 2024-09-12T16:22:03.427 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-12T16:22:03.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-12T16:22:03.428 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-12T16:22:03.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:22:03.428+0000 7f97bd3c7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 44996) UID: 1000 2024-09-12T16:22:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_13 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-12T16:22:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-12T16:22:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:22:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_13 2024-09-12T16:22:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:22:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-12T16:22:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-12T16:22:04.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:04.682+0000 7fe49c881640 -1 rbd::mirror::ImageMap: 0x55e0609dc7e0 handle_peer_ack_remove: failed to remove global_image_id=02a2eb1f-5638-48d4-814e-2b3a49935a89 2024-09-12T16:22:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_14 2024-09-12T16:22:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-12T16:22:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:22:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-12T16:22:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-12T16:22:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:22:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_14 2024-09-12T16:22:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:22:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-12T16:22:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-12T16:22:07.713 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:07.712+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c51e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:07.714 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:07.712+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177b400 [3/4d0d2df1-b0b8-4abb-b611-0cc3a70369d4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:09.429 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-12T16:22:09.429 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:22:09.429 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-12T16:22:09.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-12T16:22:09.430 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-12T16:22:09.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-12T16:22:09.433 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-12T16:22:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_15 2024-09-12T16:22:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-12T16:22:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:22:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-12T16:22:09.707 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.704+0000 7fbc5a793640 -1 reset not still connected to 0x55f82c87fba0 2024-09-12T16:22:09.707 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.704+0000 7fbc5a793640 -1 reset not still connected to 0x55f830c98820 2024-09-12T16:22:09.707 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.704+0000 7fbc5a793640 -1 reset not still connected to 0x55f830cc1930 2024-09-12T16:22:09.710 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.706+0000 7fbc5a793640 -1 reset not still connected to 0x55f83a610750 2024-09-12T16:22:09.714 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f83a610a90 2024-09-12T16:22:09.714 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f83a611790 2024-09-12T16:22:09.714 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f841b8bc70 2024-09-12T16:22:09.714 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f85b9fc270 2024-09-12T16:22:09.714 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f85b9fcd00 2024-09-12T16:22:09.714 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f85e702000 2024-09-12T16:22:09.714 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f85e702b60 2024-09-12T16:22:09.715 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f862754410 2024-09-12T16:22:09.715 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:22:09.712+0000 7fbc5a793640 -1 reset not still connected to 0x55f862755a00 2024-09-12T16:22:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-12T16:22:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:22:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_15 2024-09-12T16:22:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:22:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-12T16:22:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-12T16:22:10.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:10.724+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c5380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:10.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:10.724+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e061295b80 [3/1961696a-f14e-44a0-a222-ad6e28c429fe] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_16 2024-09-12T16:22:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-12T16:22:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:22:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-12T16:22:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-12T16:22:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:22:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_16 2024-09-12T16:22:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:22:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-12T16:22:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-12T16:22:13.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:13.739+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c5520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:13.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:13.739+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e06177b180 [3/47d38d69-792b-405f-a65b-101acd193fb9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_17 2024-09-12T16:22:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-12T16:22:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:22:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-12T16:22:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-12T16:22:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:22:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_17 2024-09-12T16:22:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:22:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-12T16:22:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-12T16:22:16.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:16.754+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0617fc820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:16.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:16.754+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177ac80 [3/3d399251-fa2f-40f2-9b35-dfd5e3c5d326] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_18 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-12T16:22:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-12T16:22:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:22:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_18 2024-09-12T16:22:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:22:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-12T16:22:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-12T16:22:19.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:19.682+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0617fc9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:19.683 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:19.682+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616be280 [3/77378832-95d1-4af4-a433-6685e1d5dd58] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_19 2024-09-12T16:22:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-12T16:22:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:22:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-12T16:22:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-12T16:22:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:22:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_19 2024-09-12T16:22:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:22:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-12T16:22:22.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-12T16:22:22.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:22.766+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c56c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:22.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:22.766+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e06177a500 [3/2152ea11-154f-4401-9170-49378499a08c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_20 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-12T16:22:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-12T16:22:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:22:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_20 2024-09-12T16:22:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:22:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-12T16:22:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-12T16:22:25.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:25.774+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c5860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:25.776 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:25.774+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616bf400 [3/ac9510df-01cf-4e79-97cb-6868ae398ef5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_21 2024-09-12T16:22:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-12T16:22:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:22:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-12T16:22:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-12T16:22:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:22:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_21 2024-09-12T16:22:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:22:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-12T16:22:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-12T16:22:28.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:28.998+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c5a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:28.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:28.998+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616bf180 [3/a8bf7886-b7c1-43a8-820b-159e65038b93] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_22 2024-09-12T16:22:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-12T16:22:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:22:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-12T16:22:30.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.29117882205249934) > weight (0.25) 2024-09-12T16:22:30.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9400153751359905) > weight (0.25) 2024-09-12T16:22:30.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9591377265688639) > weight (0.25) 2024-09-12T16:22:30.710 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.365417134676174) > weight (0.25) 2024-09-12T16:22:30.710 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-12T16:22:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-12T16:22:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:22:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_22 2024-09-12T16:22:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:22:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-12T16:22:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-12T16:22:31.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:31.690+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c5ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:31.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:31.690+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e061295180 [3/051e4edc-7365-4ada-b234-2ea4450f9d6d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_23 2024-09-12T16:22:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-12T16:22:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:22:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-12T16:22:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-12T16:22:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:22:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_23 2024-09-12T16:22:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:22:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-12T16:22:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-12T16:22:34.699 INFO:tasks.ceph.cluster1.osd.2.smithi016.stderr:2024-09-12T16:22:34.697+0000 7fd5a4fdf640 -1 reset not still connected to 0x5558cbf4c1a0 2024-09-12T16:22:34.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:34.698+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0609eba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:34.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:34.698+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e0617ddb80 [3/d6fec4d2-a64c-49a2-811d-33983c8062c0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_24 2024-09-12T16:22:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-12T16:22:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:22:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-12T16:22:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-12T16:22:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:22:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_24 2024-09-12T16:22:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:22:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-12T16:22:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-12T16:22:37.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:37.720+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c49c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:37.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:37.720+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e06177aa00 [3/31fe714a-6461-4cbe-82ca-613754e1fba5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_25 2024-09-12T16:22:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-12T16:22:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:22:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-12T16:22:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-12T16:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_25 2024-09-12T16:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:22:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-12T16:22:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-12T16:22:40.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:40.799+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c5040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:40.800 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:40.799+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0617fe500 [3/e4321129-6277-4549-af49-32f7bc842699] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_26 2024-09-12T16:22:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-12T16:22:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:22:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-12T16:22:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-12T16:22:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:22:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_26 2024-09-12T16:22:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:22:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-12T16:22:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-12T16:22:43.753 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:43.753+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c41a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:43.754 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:43.753+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0617fe000 [3/de6e82d5-259a-465e-adb7-ef275c6429d6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_27 2024-09-12T16:22:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-12T16:22:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:22:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-12T16:22:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-12T16:22:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:22:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_27 2024-09-12T16:22:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:22:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-12T16:22:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-12T16:22:46.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:46.748+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e060b1aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:46.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:46.748+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177a780 [3/2e3a6a3e-2072-4093-baab-81987b90abbb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_28 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-12T16:22:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-12T16:22:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:22:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_28 2024-09-12T16:22:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:22:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-12T16:22:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-12T16:22:49.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:49.696+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0609eb6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:49.697 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:49.696+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e061294f00 [3/04c258ba-62b0-4f2a-9ccc-60bd5563c0a1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_29 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-12T16:22:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-12T16:22:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:22:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_29 2024-09-12T16:22:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:22:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-12T16:22:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-12T16:22:52.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:52.764+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e060b1ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:52.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:52.764+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616bec80 [3/9d01e9e8-f583-4610-9203-3eeb61dc6505] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_30 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-12T16:22:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-12T16:22:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:22:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:55.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_30 2024-09-12T16:22:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:22:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-12T16:22:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-12T16:22:55.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:55.797+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e060b1ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:55.798 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:55.797+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177af00 [3/44d6b71b-d3f9-4302-8b48-4593d4e59e2c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:22:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:22:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_31 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-12T16:22:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-12T16:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:22:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_31 2024-09-12T16:22:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:22:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:22:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:22:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-12T16:22:58.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-12T16:22:58.805 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:58.804+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061317ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:22:58.806 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:22:58.804+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e06177b680 [3/5666a566-1e64-44f3-803d-958a51c3829a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_32 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-12T16:23:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-12T16:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_32 2024-09-12T16:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-12T16:23:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-12T16:23:01.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:01.695+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061316820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:01.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:01.695+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177bb80 [3/6d745883-fe82-4878-8b05-e9925084374f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_33 2024-09-12T16:23:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-12T16:23:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:23:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-12T16:23:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-12T16:23:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:23:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_33 2024-09-12T16:23:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:23:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-12T16:23:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-12T16:23:04.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:04.695+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061317a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:04.696 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:04.695+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e061295400 [3/0817148d-e3ac-4c66-9784-73a94988f707] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_34 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-12T16:23:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-12T16:23:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:23:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_34 2024-09-12T16:23:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:23:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-12T16:23:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-12T16:23:07.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:07.714+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061317380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:07.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:07.714+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177a000 [3/1cffc7cb-c521-45a9-965f-d5265ebeee09] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_35 2024-09-12T16:23:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-12T16:23:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:23:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-12T16:23:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-12T16:23:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:23:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_35 2024-09-12T16:23:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:23:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-12T16:23:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-12T16:23:10.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:10.733+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061316000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:10.735 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:10.733+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616be780 [3/82a2b8c9-597b-4a1a-9ca6-24a0c920fc52] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_36 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-12T16:23:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-12T16:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_36 2024-09-12T16:23:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:23:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-12T16:23:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-12T16:23:13.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:13.753+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061317860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:13.756 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:13.754+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e0616bef00 [3/a0349dfc-0d36-4ede-9020-d1b706f0f9dc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_37 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-12T16:23:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-12T16:23:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:23:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_37 2024-09-12T16:23:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:23:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-12T16:23:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-12T16:23:16.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:16.753+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061316ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:16.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:16.753+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e061295680 [3/0b18fcd6-90f2-4536-82ab-2679001c84e1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:17.434 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.587787485772345) > weight (0.25) 2024-09-12T16:23:17.434 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-12T16:23:17.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-12T16:23:17.435 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-12T16:23:17.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:23:17.434+0000 7f20ef712640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 61801) UID: 1000 2024-09-12T16:23:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_38 2024-09-12T16:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-12T16:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:18.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-12T16:23:19.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-12T16:23:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:23:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_38 2024-09-12T16:23:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:23:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-12T16:23:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-12T16:23:19.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:19.702+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0613171e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:19.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:19.702+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0617fe280 [3/e0e6fdb8-ef78-418e-9cc0-4afd4bd69e5b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:21.436 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-12T16:23:21.436 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:23:21.437 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-12T16:23:21.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-12T16:23:21.437 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-12T16:23:21.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-12T16:23:21.440 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-12T16:23:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_39 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-12T16:23:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-12T16:23:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:23:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_39 2024-09-12T16:23:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:23:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-12T16:23:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-12T16:23:22.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:22.980+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c4000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:22.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:22.980+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0617dc280 [3/d0fe3ef0-64f0-4fc2-be1f-5b61839e3452] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_40 2024-09-12T16:23:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-12T16:23:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:23:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-12T16:23:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-12T16:23:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:23:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_40 2024-09-12T16:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:23:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-12T16:23:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-12T16:23:25.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:25.791+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e060b1aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:25.792 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:25.791+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616bf900 [3/b72eb11e-35a3-4497-848c-fa02d50ba2c9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_41 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:27.725 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-12T16:23:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-12T16:23:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:23:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_41 2024-09-12T16:23:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:23:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-12T16:23:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-12T16:23:28.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:28.810+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0609eba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:28.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:28.810+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177a280 [3/208f0c1a-c137-477d-8165-0c0b6213b01f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_42 2024-09-12T16:23:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-12T16:23:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:23:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:30.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-12T16:23:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-12T16:23:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:23:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_42 2024-09-12T16:23:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:23:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-12T16:23:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-12T16:23:31.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:31.703+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0616c5d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:31.704 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:31.703+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e061295900 [3/13c21475-07e1-46bd-818c-3e23de114deb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:33.617 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:23:33.616+0000 7fbc5a793640 -1 reset not still connected to 0x55f862754680 2024-09-12T16:23:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_43 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-12T16:23:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-12T16:23:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:23:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_43 2024-09-12T16:23:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:23:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-12T16:23:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-12T16:23:34.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:34.707+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0609eb520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:34.708 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:34.707+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e06177b900 [3/5e72a01d-2edd-4c16-ac2b-2b4399cf2dd2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_44 2024-09-12T16:23:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-12T16:23:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:23:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-12T16:23:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-12T16:23:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:23:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_44 2024-09-12T16:23:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:23:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-12T16:23:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-12T16:23:37.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:37.730+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061203ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:37.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:37.730+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616be000 [3/744e74ce-448f-4d0c-b2d1-b1a3be444e01] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_45 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-12T16:23:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-12T16:23:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:23:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_45 2024-09-12T16:23:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:23:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-12T16:23:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-12T16:23:40.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:40.735+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061203a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:40.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:40.735+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616bf680 [3/ad26f91c-a3fe-4c1d-910b-f56df2175de9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:42.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_46 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-12T16:23:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-12T16:23:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:23:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_46 2024-09-12T16:23:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:23:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-12T16:23:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-12T16:23:43.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.811368495169295) > weight (0.25) 2024-09-12T16:23:43.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7461541700987775) > weight (0.25) 2024-09-12T16:23:43.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9922682127164696) > weight (0.25) 2024-09-12T16:23:43.711 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6836173468416842) > weight (0.25) 2024-09-12T16:23:43.711 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-12T16:23:43.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:43.749+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e06095ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:43.750 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:43.749+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e0617dc000 [3/c85a9f05-e741-40a3-80f4-86c92eb59802] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_47 2024-09-12T16:23:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-12T16:23:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:23:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-12T16:23:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-12T16:23:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:23:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_47 2024-09-12T16:23:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:23:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-12T16:23:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-12T16:23:46.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:46.760+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061317860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:46.761 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:46.760+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0617fe780 [3/eac71c3a-77dc-48b2-a8e3-c7000c1abbd9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_48 2024-09-12T16:23:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-12T16:23:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:23:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-12T16:23:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-12T16:23:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:23:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_48 2024-09-12T16:23:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:23:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-12T16:23:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-12T16:23:49.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:49.719+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061317040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:49.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:49.719+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616bea00 [3/9815c7d5-2e9a-44c3-b4bc-6316340f81e4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_49 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-12T16:23:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-12T16:23:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:23:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_49 2024-09-12T16:23:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:23:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-12T16:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-12T16:23:52.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:52.790+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e06101bd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:52.791 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:52.790+0000 7fe49c080640 -1 rbd::mirror::ImageReplayer: 0x55e0617dc500 [3/cb8151fa-d57f-4d3c-8f0f-b2e94e60e773] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_50 2024-09-12T16:23:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-12T16:23:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:23:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local snap=snap 2024-09-12T16:23:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-12T16:23:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing snap: 100% complete...done. 2024-09-12T16:23:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-12T16:23:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:23:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:23:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_50 2024-09-12T16:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-12T16:23:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-12T16:23:55.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:55.803+0000 7fe496074640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e061316b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:23:55.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:23:55.803+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0616be500 [3/7e4292a4-067e-49ff-9c07-bb55509086c8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:23:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-12T16:23:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-12T16:23:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ tee -a /tmp/tmp.65UL20DnMX/rbd-mirror.test.log 2024-09-12T16:23:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ date '+%F %T' 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo 2024-09-12 16:23:57 TEST: image deletions should propagate 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:2024-09-12 16:23:57 TEST: image deletions should propagate 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_count=0 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local count 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local last_count=0 2024-09-12T16:23:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ true 2024-09-12T16:23:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 40 2024-09-12T16:23:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in `seq 1 40` 2024-09-12T16:23:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 -ne 1 2024-09-12T16:23:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T16:23:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep 'images: ' 2024-09-12T16:23:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cut '-d ' -f 2 2024-09-12T16:23:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T16:23:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ count=0 2024-09-12T16:23:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 = 0 2024-09-12T16:23:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-12T16:23:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state 2024-09-12T16:23:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 40 2024-09-12T16:23:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in `seq 1 40` 2024-09-12T16:23:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 -ne 1 2024-09-12T16:23:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-12T16:23:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep 'image health:' 2024-09-12T16:23:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cut '-d ' -f 3 2024-09-12T16:23:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: failed to query services: (13) Permission denied 2024-09-12T16:23:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ state=OK 2024-09-12T16:23:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test OK = ERROR 2024-09-12T16:23:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test OK = OK 2024-09-12T16:23:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 50 2024-09-12T16:23:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_1 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T16:23:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_1 2024-09-12T16:23:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T16:23:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T16:23:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T16:23:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_1 2024-09-12T16:23:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T16:23:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_2 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_2 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T16:23:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T16:23:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T16:23:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_2 2024-09-12T16:23:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T16:23:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T16:23:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_3 2024-09-12T16:23:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_3 2024-09-12T16:23:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T16:23:58.209 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T16:23:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T16:23:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_3 2024-09-12T16:23:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T16:23:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T16:23:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_4 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T16:23:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_4 2024-09-12T16:23:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T16:23:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T16:23:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T16:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_4 2024-09-12T16:23:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T16:23:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_5 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:23:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_5 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T16:23:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T16:23:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-12T16:23:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:23:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_5 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T16:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T16:23:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_6 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T16:23:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_6 2024-09-12T16:23:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T16:23:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T16:23:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_6 2024-09-12T16:23:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T16:23:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_7 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_7 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T16:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T16:23:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-12T16:23:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:23:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_7 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T16:23:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T16:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_8 2024-09-12T16:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-12T16:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T16:23:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_8 2024-09-12T16:23:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T16:23:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T16:23:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-12T16:23:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:23:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_8 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T16:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T16:23:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_9 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_9 2024-09-12T16:23:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T16:23:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T16:23:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_9 2024-09-12T16:23:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T16:23:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_10 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_10 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T16:23:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T16:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_10 2024-09-12T16:23:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T16:23:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T16:23:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_11 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_11 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_11 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-12T16:23:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-12T16:23:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:23:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-12T16:23:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_11 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_11 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_11 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-12T16:23:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-12T16:23:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:23:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_12 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:23:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:23:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_12 2024-09-12T16:23:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:23:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:23:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_12 2024-09-12T16:23:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-12T16:23:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:23:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-12T16:23:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:23:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:23:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_12 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_12 2024-09-12T16:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_12 2024-09-12T16:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-12T16:24:00.002 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_13 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_13 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_13 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-12T16:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-12T16:24:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-12T16:24:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_13 2024-09-12T16:24:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_13 2024-09-12T16:24:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_13 2024-09-12T16:24:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-12T16:24:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-12T16:24:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_14 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_14 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_14 2024-09-12T16:24:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-12T16:24:00.225 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-12T16:24:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_14 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_14 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_14 2024-09-12T16:24:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-12T16:24:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-12T16:24:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_15 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_15 2024-09-12T16:24:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_15 2024-09-12T16:24:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-12T16:24:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-12T16:24:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_15 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_15 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_15 2024-09-12T16:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-12T16:24:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-12T16:24:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_16 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_16 2024-09-12T16:24:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_16 2024-09-12T16:24:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-12T16:24:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-12T16:24:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-12T16:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_16 2024-09-12T16:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_16 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_16 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-12T16:24:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-12T16:24:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_17 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_17 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_17 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-12T16:24:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-12T16:24:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-12T16:24:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_17 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_17 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_17 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-12T16:24:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-12T16:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_18 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_18 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_18 2024-09-12T16:24:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-12T16:24:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-12T16:24:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-12T16:24:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_18 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_18 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_18 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-12T16:24:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_19 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_19 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_19 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-12T16:24:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-12T16:24:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_19 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_19 2024-09-12T16:24:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_19 2024-09-12T16:24:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-12T16:24:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_20 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-12T16:24:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_20 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_20 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-12T16:24:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-12T16:24:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-12T16:24:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_20 2024-09-12T16:24:01.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_20 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_20 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-12T16:24:01.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_21 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-12T16:24:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_21 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_21 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-12T16:24:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-12T16:24:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-12T16:24:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_21 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_21 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_21 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-12T16:24:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_22 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:24:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_22 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_22 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-12T16:24:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-12T16:24:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_22 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_22 2024-09-12T16:24:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_22 2024-09-12T16:24:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-12T16:24:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-12T16:24:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_23 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_23 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_23 2024-09-12T16:24:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-12T16:24:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-12T16:24:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-12T16:24:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_23 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_23 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_23 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-12T16:24:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-12T16:24:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_24 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_24 2024-09-12T16:24:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_24 2024-09-12T16:24:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-12T16:24:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-12T16:24:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-12T16:24:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_24 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_24 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_24 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-12T16:24:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_25 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:24:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_25 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_25 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-12T16:24:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-12T16:24:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-12T16:24:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_25 2024-09-12T16:24:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_25 2024-09-12T16:24:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_25 2024-09-12T16:24:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-12T16:24:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_26 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_26 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_26 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-12T16:24:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.448 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-12T16:24:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-12T16:24:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_26 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_26 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_26 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-12T16:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-12T16:24:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_27 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_27 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_27 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-12T16:24:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-12T16:24:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-12T16:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_27 2024-09-12T16:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_27 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_27 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-12T16:24:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_28 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-12T16:24:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_28 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_28 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-12T16:24:02.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-12T16:24:02.818 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-12T16:24:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_28 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_28 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_28 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-12T16:24:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-12T16:24:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_29 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_29 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_29 2024-09-12T16:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-12T16:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-12T16:24:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-12T16:24:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_29 2024-09-12T16:24:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_29 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_29 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-12T16:24:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.139 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_30 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-12T16:24:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_30 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_30 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-12T16:24:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-12T16:24:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_30 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_30 2024-09-12T16:24:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_30 2024-09-12T16:24:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-12T16:24:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_31 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_31 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_31 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-12T16:24:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-12T16:24:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_31 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_31 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_31 2024-09-12T16:24:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-12T16:24:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-12T16:24:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_32 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_32 2024-09-12T16:24:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_32 2024-09-12T16:24:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-12T16:24:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-12T16:24:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:03.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:03.552 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_32 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_32 2024-09-12T16:24:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_32 2024-09-12T16:24:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-12T16:24:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-12T16:24:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_33 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_33 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_33 2024-09-12T16:24:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-12T16:24:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-12T16:24:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_33 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_33 2024-09-12T16:24:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_33 2024-09-12T16:24:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-12T16:24:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-12T16:24:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_34 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_34 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_34 2024-09-12T16:24:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-12T16:24:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-12T16:24:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-12T16:24:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_34 2024-09-12T16:24:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_34 2024-09-12T16:24:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_34 2024-09-12T16:24:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-12T16:24:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-12T16:24:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_35 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_35 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_35 2024-09-12T16:24:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-12T16:24:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-12T16:24:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_35 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_35 2024-09-12T16:24:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_35 2024-09-12T16:24:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-12T16:24:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-12T16:24:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_36 2024-09-12T16:24:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_36 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_36 2024-09-12T16:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-12T16:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-12T16:24:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_36 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_36 2024-09-12T16:24:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_36 2024-09-12T16:24:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-12T16:24:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-12T16:24:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_37 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_37 2024-09-12T16:24:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_37 2024-09-12T16:24:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-12T16:24:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-12T16:24:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-12T16:24:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_37 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_37 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_37 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-12T16:24:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_38 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_38 2024-09-12T16:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_38 2024-09-12T16:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-12T16:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-12T16:24:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-12T16:24:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_38 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_38 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_38 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-12T16:24:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_39 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_39 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_39 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-12T16:24:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-12T16:24:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:05.044 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-12T16:24:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.146 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_39 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_39 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_39 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-12T16:24:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-12T16:24:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_40 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_40 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_40 2024-09-12T16:24:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-12T16:24:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-12T16:24:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-12T16:24:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_40 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_40 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_40 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-12T16:24:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-12T16:24:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_41 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_41 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_41 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-12T16:24:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-12T16:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_41 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_41 2024-09-12T16:24:05.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_41 2024-09-12T16:24:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-12T16:24:05.513 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_42 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_42 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_42 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-12T16:24:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-12T16:24:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-12T16:24:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_42 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_42 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_42 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-12T16:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-12T16:24:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_43 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_43 2024-09-12T16:24:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_43 2024-09-12T16:24:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-12T16:24:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-12T16:24:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_43 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_43 2024-09-12T16:24:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_43 2024-09-12T16:24:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-12T16:24:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-12T16:24:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_44 2024-09-12T16:24:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_44 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_44 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-12T16:24:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-12T16:24:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-12T16:24:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_44 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_44 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_44 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-12T16:24:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_45 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:24:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_45 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_45 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-12T16:24:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-12T16:24:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_45 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_45 2024-09-12T16:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_45 2024-09-12T16:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-12T16:24:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-12T16:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_46 2024-09-12T16:24:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_46 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_46 2024-09-12T16:24:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-12T16:24:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-12T16:24:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-12T16:24:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_46 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_46 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_46 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-12T16:24:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_47 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-12T16:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_47 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_47 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-12T16:24:06.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-12T16:24:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_47 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_47 2024-09-12T16:24:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_47 2024-09-12T16:24:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-12T16:24:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-12T16:24:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_48 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_48 2024-09-12T16:24:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_48 2024-09-12T16:24:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-12T16:24:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-12T16:24:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-12T16:24:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_48 2024-09-12T16:24:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_48 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_48 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-12T16:24:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-12T16:24:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_49 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_49 2024-09-12T16:24:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_49 2024-09-12T16:24:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-12T16:24:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:06.865 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-12T16:24:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_49 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_49 2024-09-12T16:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_49 2024-09-12T16:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-12T16:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-12T16:24:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:07.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_50 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_50 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_50 2024-09-12T16:24:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-12T16:24:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-12T16:24:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_50 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_50 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_50 2024-09-12T16:24:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-12T16:24:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-12T16:24:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-12T16:24:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ tee -a /tmp/tmp.65UL20DnMX/rbd-mirror.test.log 2024-09-12T16:24:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ date '+%F %T' 2024-09-12T16:24:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo 2024-09-12 16:24:07 TEST: delete images during bootstrap 2024-09-12T16:24:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:2024-09-12 16:24:07 TEST: delete images during bootstrap 2024-09-12T16:24:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-12T16:24:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=image 2024-09-12T16:24:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-12T16:24:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd: mirroring is already configured for image mode with remote namespace 2024-09-12T16:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-12T16:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=image 2024-09-12T16:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-12T16:24:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd: mirroring is already configured for image mode with remote namespace 2024-09-12T16:24:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ start_mirror cluster1 2024-09-12T16:24:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance 2024-09-12T16:24:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local val=cluster1 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster_var_name=cluster 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local instance_var_name=instance 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cluster=cluster1 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=cluster1 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ instance=0 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval cluster=cluster1 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ cluster=cluster1 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ eval instance=0 2024-09-12T16:24:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ instance=0 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=test 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 1 10 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_1 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:24:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:24:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:24:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-12T16:24:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-12T16:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-12T16:24:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-12T16:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_1': 2024-09-12T16:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 18f84648c9da 2024-09-12T16:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.18f84648c9da 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:24:07 2024 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:24:07 2024 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:24:07 2024 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 9e9e02b3-181b-4ca6-b6aa-5d6fc306cee5 2024-09-12T16:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-12T16:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=13 2024-09-12T16:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:24:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 34459648 bytes 2024-09-12T16:24:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 43276288 bytes 2024-09-12T16:24:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 111149056 bytes 2024-09-12T16:24:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 126078976 bytes 2024-09-12T16:24:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 141630464 bytes 2024-09-12T16:24:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 157792256 bytes 2024-09-12T16:24:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 177336320 bytes 2024-09-12T16:24:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 218894336 bytes 2024-09-12T16:24:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 229798912 bytes 2024-09-12T16:24:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 234465280 bytes 2024-09-12T16:24:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 246410240 bytes 2024-09-12T16:24:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 271173632 bytes 2024-09-12T16:24:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 308861952 bytes 2024-09-12T16:24:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 369277952 bytes 2024-09-12T16:24:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 434044928 bytes 2024-09-12T16:24:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 464177152 bytes 2024-09-12T16:24:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 506903552 bytes 2024-09-12T16:24:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 541956096 bytes 2024-09-12T16:24:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 592693248 bytes 2024-09-12T16:24:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 604004352 bytes 2024-09-12T16:24:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 651309056 bytes 2024-09-12T16:24:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 717252608 bytes 2024-09-12T16:24:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 729717760 bytes 2024-09-12T16:24:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 792181760 bytes 2024-09-12T16:24:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 792803328 bytes 2024-09-12T16:24:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 837163008 bytes 2024-09-12T16:24:11.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 875969536 bytes 2024-09-12T16:24:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 926208000 bytes 2024-09-12T16:24:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 968619008 bytes 2024-09-12T16:24:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1030711296 bytes 2024-09-12T16:24:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1067952128 bytes 2024-09-12T16:24:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1085527040 bytes 2024-09-12T16:24:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1126072320 bytes 2024-09-12T16:24:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1200921600 bytes 2024-09-12T16:24:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1264194560 bytes 2024-09-12T16:24:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1310289920 bytes 2024-09-12T16:24:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1372032000 bytes 2024-09-12T16:24:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1400424448 bytes 2024-09-12T16:24:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1424069632 bytes 2024-09-12T16:24:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1450622976 bytes 2024-09-12T16:24:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1500925952 bytes 2024-09-12T16:24:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1545816064 bytes 2024-09-12T16:24:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1572131840 bytes 2024-09-12T16:24:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1606806528 bytes 2024-09-12T16:24:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1667189760 bytes 2024-09-12T16:24:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1681210368 bytes 2024-09-12T16:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1699764224 bytes 2024-09-12T16:24:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1715657728 bytes 2024-09-12T16:24:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1717389312 bytes 2024-09-12T16:24:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1737582592 bytes 2024-09-12T16:24:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 1809090560 bytes 2024-09-12T16:24:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 1833403392 bytes 2024-09-12T16:24:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 1908373504 bytes 2024-09-12T16:24:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 1918156800 bytes 2024-09-12T16:24:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 1960371200 bytes 2024-09-12T16:24:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2022026240 bytes 2024-09-12T16:24:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2064802816 bytes 2024-09-12T16:24:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2095120384 bytes 2024-09-12T16:24:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2174944256 bytes 2024-09-12T16:24:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2242738176 bytes 2024-09-12T16:24:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2266457088 bytes 2024-09-12T16:24:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2323260416 bytes 2024-09-12T16:24:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2329927680 bytes 2024-09-12T16:24:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2373257216 bytes 2024-09-12T16:24:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 2398577664 bytes 2024-09-12T16:24:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 2437332992 bytes 2024-09-12T16:24:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 2477771776 bytes 2024-09-12T16:24:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 2482701312 bytes 2024-09-12T16:24:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 2567017472 bytes 2024-09-12T16:24:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 2670248960 bytes 2024-09-12T16:24:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 2738442240 bytes 2024-09-12T16:24:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 2758746112 bytes 2024-09-12T16:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 2780801024 bytes 2024-09-12T16:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 2806021120 bytes 2024-09-12T16:24:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 2828374016 bytes 2024-09-12T16:24:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7600 IOs, 2852796416 bytes 2024-09-12T16:24:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7700 IOs, 2909097984 bytes 2024-09-12T16:24:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_1 2024-09-12T16:24:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T16:24:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b2a13295f 2024-09-12T16:24:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_1 present 128b2a13295f 2024-09-12T16:24:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:24:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b2a13295f 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_1 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T16:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b2a13295f 2024-09-12T16:24:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b2a13295f 2024-09-12T16:24:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b2a13295f -o 128b2a13295f = 128b2a13295f 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-12T16:24:33.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:24:33.543+0000 7fe495873640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e0613176c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:24:33.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:24:33.543+0000 7fe49c881640 -1 rbd::mirror::ImageReplayer: 0x55e0617dc500 [3/9e9e02b3-181b-4ca6-b6aa-5d6fc306cee5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:24:51.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:24:51.422+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b49cec00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-12T16:24:51.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:24:51.422+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b49cec00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=185, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-12T16:24:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:24:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-12T16:24:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_1 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T16:24:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T16:24:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:24:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_1 2024-09-12T16:24:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_1 2024-09-12T16:24:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-12T16:24:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-12T16:24:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-12T16:24:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:24:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:24:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_1 2024-09-12T16:24:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_1 2024-09-12T16:24:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-12T16:24:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-12T16:24:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_2 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:24:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:24:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:24:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:24:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:24:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-12T16:24:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-12T16:24:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:24:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:24:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-12T16:24:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:24:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_2': 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 191061791f0d 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.191061791f0d 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:24:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:24:52 2024 2024-09-12T16:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:24:52 2024 2024-09-12T16:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:24:52 2024 2024-09-12T16:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 22e6b9de-360e-4ddf-81d0-342820f41f00 2024-09-12T16:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:24:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-12T16:24:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:24:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:24:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=7 2024-09-12T16:24:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:24:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:24:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 17810432 bytes 2024-09-12T16:24:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 101509120 bytes 2024-09-12T16:24:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 149381120 bytes 2024-09-12T16:24:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 161756160 bytes 2024-09-12T16:24:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 205563904 bytes 2024-09-12T16:24:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 240087040 bytes 2024-09-12T16:24:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 313548800 bytes 2024-09-12T16:24:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 362983424 bytes 2024-09-12T16:24:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 426887168 bytes 2024-09-12T16:24:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 442774528 bytes 2024-09-12T16:24:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 502833152 bytes 2024-09-12T16:24:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 503453696 bytes 2024-09-12T16:24:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 523656192 bytes 2024-09-12T16:24:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 618979328 bytes 2024-09-12T16:24:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 716832768 bytes 2024-09-12T16:24:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 752028672 bytes 2024-09-12T16:24:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 789289984 bytes 2024-09-12T16:24:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 874170368 bytes 2024-09-12T16:24:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 932953088 bytes 2024-09-12T16:24:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 952623104 bytes 2024-09-12T16:24:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 979993600 bytes 2024-09-12T16:24:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 984000512 bytes 2024-09-12T16:24:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1056124928 bytes 2024-09-12T16:24:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1087004672 bytes 2024-09-12T16:24:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1110806528 bytes 2024-09-12T16:24:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1132701696 bytes 2024-09-12T16:24:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1169359872 bytes 2024-09-12T16:24:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1216323584 bytes 2024-09-12T16:24:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1225694208 bytes 2024-09-12T16:24:57.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1285793792 bytes 2024-09-12T16:24:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1331943424 bytes 2024-09-12T16:24:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1387242496 bytes 2024-09-12T16:24:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1463350272 bytes 2024-09-12T16:24:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1478921216 bytes 2024-09-12T16:24:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1489845248 bytes 2024-09-12T16:24:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1543723008 bytes 2024-09-12T16:24:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1608128512 bytes 2024-09-12T16:24:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1614766080 bytes 2024-09-12T16:24:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1726330880 bytes 2024-09-12T16:24:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:24:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:24:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:24:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:24:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T16:24:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_2 2024-09-12T16:24:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T16:24:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b7face8d7 2024-09-12T16:24:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:24:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_2 present 128b7face8d7 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b7face8d7 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:24:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T16:24:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:24:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:24:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_2 2024-09-12T16:24:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T16:24:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:24:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b7face8d7 2024-09-12T16:24:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b7face8d7 2024-09-12T16:24:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b7face8d7 -o 128b7face8d7 = 128b7face8d7 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:24:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-12T16:25:00.442 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5670165377774988) > weight (0.25) 2024-09-12T16:25:00.442 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8656949880162672) > weight (0.25) 2024-09-12T16:25:00.442 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2592918330058245) > weight (0.25) 2024-09-12T16:25:00.442 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-12T16:25:00.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-12T16:25:00.443 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-12T16:25:00.443 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:25:00.443+0000 7fe49d082640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 59575) UID: 1000 2024-09-12T16:25:04.509 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:25:04.507+0000 7f92f9dd9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6d66a1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:25:04.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:25:04.507+0000 7f92ffde5640 -1 rbd::mirror::ImageReplayer: 0x557c6c29e280 [3/22e6b9de-360e-4ddf-81d0-342820f41f00] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:25:06.444 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-12T16:25:06.444 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:25:06.444 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-12T16:25:06.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-12T16:25:06.445 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-12T16:25:06.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-12T16:25:06.448 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-12T16:25:13.712 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T16:25:13.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T16:25:13.713 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-12T16:25:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:25:13.713+0000 7fa33ea26640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 60995) UID: 1000 2024-09-12T16:25:21.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:25:21.601+0000 7f92f9dd9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6d66a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:25:21.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:25:21.601+0000 7f92ffde5640 -1 rbd::mirror::ImageReplayer: 0x557c6c29e280 [3/22e6b9de-360e-4ddf-81d0-342820f41f00] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:25:22.714 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T16:25:22.714 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:25:22.715 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T16:25:22.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T16:25:22.715 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T16:25:22.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T16:25:22.719 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-12T16:25:29.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:25:29.546+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b1cb9400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-12T16:25:29.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:25:29.546+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b1cb9400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=186, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-12T16:25:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:25:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-12T16:25:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:25:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:25:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_2 2024-09-12T16:25:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T16:25:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:25:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T16:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:25:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_2 2024-09-12T16:25:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:25:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:25:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_2 2024-09-12T16:25:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-12T16:25:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:25:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:25:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-12T16:25:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-12T16:25:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:25:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:25:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:25:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:25:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_2 2024-09-12T16:25:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:25:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_2 2024-09-12T16:25:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-12T16:25:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-12T16:25:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_3 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:25:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:25:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:25:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:25:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-12T16:25:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-12T16:25:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:25:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:25:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:25:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-12T16:25:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:25:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-12T16:25:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_3': 2024-09-12T16:25:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:25:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 1937e79b3365 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.1937e79b3365 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:25:30 2024 2024-09-12T16:25:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:25:30 2024 2024-09-12T16:25:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:25:30 2024 2024-09-12T16:25:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:25:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:25:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 02dea27c-61c9-469a-ba33-a0140b450d00 2024-09-12T16:25:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:25:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-12T16:25:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:25:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:25:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:25:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:25:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:25:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:25:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 22366208 bytes 2024-09-12T16:25:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 96415744 bytes 2024-09-12T16:25:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 108492800 bytes 2024-09-12T16:25:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 158338048 bytes 2024-09-12T16:25:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 168744960 bytes 2024-09-12T16:25:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 203534336 bytes 2024-09-12T16:25:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 251862016 bytes 2024-09-12T16:25:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 273781760 bytes 2024-09-12T16:25:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 328984576 bytes 2024-09-12T16:25:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 397788160 bytes 2024-09-12T16:25:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 424015872 bytes 2024-09-12T16:25:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 459183104 bytes 2024-09-12T16:25:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 510167040 bytes 2024-09-12T16:25:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 544308224 bytes 2024-09-12T16:25:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 569290752 bytes 2024-09-12T16:25:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 571278336 bytes 2024-09-12T16:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 605608960 bytes 2024-09-12T16:25:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 680036352 bytes 2024-09-12T16:25:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 710072320 bytes 2024-09-12T16:25:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 781799424 bytes 2024-09-12T16:25:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 840433664 bytes 2024-09-12T16:25:34.269 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 875299840 bytes 2024-09-12T16:25:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 906720256 bytes 2024-09-12T16:25:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 948401152 bytes 2024-09-12T16:25:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 964981760 bytes 2024-09-12T16:25:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1000879104 bytes 2024-09-12T16:25:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1021340672 bytes 2024-09-12T16:25:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1060004864 bytes 2024-09-12T16:25:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1141072896 bytes 2024-09-12T16:25:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1175013376 bytes 2024-09-12T16:25:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:25:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:25:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:25:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_3 2024-09-12T16:25:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T16:25:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:25:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b12e729a5 2024-09-12T16:25:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:25:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:25:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_3 present 128b12e729a5 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b12e729a5 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:25:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_3 2024-09-12T16:25:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T16:25:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b12e729a5 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b12e729a5 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b12e729a5 -o 128b12e729a5 = 128b12e729a5 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:25:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-12T16:25:51.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:25:51.601+0000 7f92f95d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6d66a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:25:51.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:25:51.601+0000 7f92ffde5640 -1 rbd::mirror::ImageReplayer: 0x557c6d6daa00 [3/02dea27c-61c9-469a-ba33-a0140b450d00] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:26:02.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:26:02.421+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b1ca3400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-12T16:26:02.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:26:02.421+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b1ca3400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-12T16:26:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:26:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-12T16:26:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_3 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T16:26:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T16:26:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:26:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:26:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:26:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_3 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_3 2024-09-12T16:26:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-12T16:26:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-12T16:26:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:26:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:26:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:26:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-12T16:26:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:26:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-12T16:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-12T16:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:26:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_3 2024-09-12T16:26:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_3 2024-09-12T16:26:02.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-12T16:26:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-12T16:26:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:26:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:26:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_4 2024-09-12T16:26:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-12T16:26:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-12T16:26:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-12T16:26:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:26:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-12T16:26:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:26:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-12T16:26:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_4': 2024-09-12T16:26:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:26:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:26:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:26:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 194f62f74ad9 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.194f62f74ad9 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:26:03 2024 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:26:03 2024 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:26:03 2024 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:26:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 53e89855-7f63-4a49-bacb-981ea18040d2 2024-09-12T16:26:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:26:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-12T16:26:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:26:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=10 2024-09-12T16:26:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:26:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:26:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 54855680 bytes 2024-09-12T16:26:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 118038528 bytes 2024-09-12T16:26:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 184612864 bytes 2024-09-12T16:26:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 214057984 bytes 2024-09-12T16:26:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 284791808 bytes 2024-09-12T16:26:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 303572992 bytes 2024-09-12T16:26:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 315646976 bytes 2024-09-12T16:26:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 357937152 bytes 2024-09-12T16:26:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 378604544 bytes 2024-09-12T16:26:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 437543936 bytes 2024-09-12T16:26:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 460428288 bytes 2024-09-12T16:26:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 471784448 bytes 2024-09-12T16:26:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 472375296 bytes 2024-09-12T16:26:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 551212032 bytes 2024-09-12T16:26:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 558220288 bytes 2024-09-12T16:26:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 605165568 bytes 2024-09-12T16:26:06.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 687027200 bytes 2024-09-12T16:26:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 766364672 bytes 2024-09-12T16:26:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 781378560 bytes 2024-09-12T16:26:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 812547072 bytes 2024-09-12T16:26:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 817360896 bytes 2024-09-12T16:26:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 829357056 bytes 2024-09-12T16:26:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 846495744 bytes 2024-09-12T16:26:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 906697728 bytes 2024-09-12T16:26:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 917221376 bytes 2024-09-12T16:26:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 935693312 bytes 2024-09-12T16:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1005348864 bytes 2024-09-12T16:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1028029440 bytes 2024-09-12T16:26:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1071576064 bytes 2024-09-12T16:26:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1145519104 bytes 2024-09-12T16:26:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1186504704 bytes 2024-09-12T16:26:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1245369344 bytes 2024-09-12T16:26:09.065 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1332177920 bytes 2024-09-12T16:26:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1393570816 bytes 2024-09-12T16:26:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1439393792 bytes 2024-09-12T16:26:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1491396608 bytes 2024-09-12T16:26:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1543151616 bytes 2024-09-12T16:26:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1613987840 bytes 2024-09-12T16:26:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1663579136 bytes 2024-09-12T16:26:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1705151488 bytes 2024-09-12T16:26:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1717024768 bytes 2024-09-12T16:26:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1764118528 bytes 2024-09-12T16:26:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1842249728 bytes 2024-09-12T16:26:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1867368448 bytes 2024-09-12T16:26:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1929356288 bytes 2024-09-12T16:26:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1997197312 bytes 2024-09-12T16:26:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2026334208 bytes 2024-09-12T16:26:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2058279936 bytes 2024-09-12T16:26:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2135823360 bytes 2024-09-12T16:26:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2198803456 bytes 2024-09-12T16:26:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2271642624 bytes 2024-09-12T16:26:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2361536512 bytes 2024-09-12T16:26:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2408228864 bytes 2024-09-12T16:26:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2453138432 bytes 2024-09-12T16:26:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2508248064 bytes 2024-09-12T16:26:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:26:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:26:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T16:26:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_4 2024-09-12T16:26:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T16:26:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b4e2b3096 2024-09-12T16:26:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:26:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:26:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_4 present 128b4e2b3096 2024-09-12T16:26:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b4e2b3096 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_4 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T16:26:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b4e2b3096 2024-09-12T16:26:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b4e2b3096 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b4e2b3096 -o 128b4e2b3096 = 128b4e2b3096 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-12T16:26:21.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:26:21.602+0000 7f92f9dd9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6d66b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:26:21.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:26:21.602+0000 7f92ffde5640 -1 rbd::mirror::ImageReplayer: 0x557c6d6da500 [3/53e89855-7f63-4a49-bacb-981ea18040d2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:26:32.719 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-12T16:26:32.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-12T16:26:32.720 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-12T16:26:32.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:26:32.720+0000 7f7aa0c9d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 71741) UID: 1000 2024-09-12T16:26:37.449 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-12T16:26:37.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-12T16:26:37.450 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-12T16:26:37.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:26:37.450+0000 7f6fab66c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 46469) UID: 1000 2024-09-12T16:26:37.721 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-12T16:26:37.721 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:26:37.722 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-12T16:26:37.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-12T16:26:37.722 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-12T16:26:37.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-12T16:26:37.725 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-12T16:26:40.450 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-12T16:26:40.451 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:26:40.451 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-12T16:26:40.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-12T16:26:40.451 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-12T16:26:40.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-12T16:26:40.454 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-12T16:26:40.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:26:40.585+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b2b55c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-09-12T16:26:40.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:26:40.585+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b2b55c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-09-12T16:26:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:26:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-12T16:26:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:26:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_4 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T16:26:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:40.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T16:26:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:26:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:26:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:26:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_4 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_4 2024-09-12T16:26:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-12T16:26:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-12T16:26:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-12T16:26:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:26:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:26:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_4 2024-09-12T16:26:40.859 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_4 2024-09-12T16:26:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-12T16:26:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-12T16:26:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:26:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:26:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_5 2024-09-12T16:26:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-12T16:26:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:26:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:26:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:26:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:26:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:26:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-12T16:26:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-12T16:26:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:26:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-12T16:26:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:26:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-12T16:26:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_5': 2024-09-12T16:26:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 197630a16da0 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.197630a16da0 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:26:41 2024 2024-09-12T16:26:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:26:41 2024 2024-09-12T16:26:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:26:41 2024 2024-09-12T16:26:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:26:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:26:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: f8418aca-7b96-4b9d-8453-91acf73b514b 2024-09-12T16:26:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:26:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-12T16:26:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:26:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:26:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=11 2024-09-12T16:26:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:26:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:26:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 44489728 bytes 2024-09-12T16:26:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 52303872 bytes 2024-09-12T16:26:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 117215232 bytes 2024-09-12T16:26:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 163799040 bytes 2024-09-12T16:26:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 214766592 bytes 2024-09-12T16:26:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 264241152 bytes 2024-09-12T16:26:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 280924160 bytes 2024-09-12T16:26:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 304150528 bytes 2024-09-12T16:26:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 332764160 bytes 2024-09-12T16:26:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 354997248 bytes 2024-09-12T16:26:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 420997120 bytes 2024-09-12T16:26:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 458468352 bytes 2024-09-12T16:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 476673024 bytes 2024-09-12T16:26:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 481568768 bytes 2024-09-12T16:26:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 491574272 bytes 2024-09-12T16:26:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 514178048 bytes 2024-09-12T16:26:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 540656640 bytes 2024-09-12T16:26:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 560701440 bytes 2024-09-12T16:26:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 595879936 bytes 2024-09-12T16:26:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 646283264 bytes 2024-09-12T16:26:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 685798400 bytes 2024-09-12T16:26:45.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 721630208 bytes 2024-09-12T16:26:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 749150208 bytes 2024-09-12T16:26:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 809832448 bytes 2024-09-12T16:26:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 861612032 bytes 2024-09-12T16:26:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 878930944 bytes 2024-09-12T16:26:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 924836864 bytes 2024-09-12T16:26:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 981769216 bytes 2024-09-12T16:26:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1062156288 bytes 2024-09-12T16:26:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1097610240 bytes 2024-09-12T16:26:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1149003776 bytes 2024-09-12T16:26:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1180756992 bytes 2024-09-12T16:26:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1250523136 bytes 2024-09-12T16:26:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1316767744 bytes 2024-09-12T16:26:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1380188160 bytes 2024-09-12T16:26:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1380812800 bytes 2024-09-12T16:26:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1442739200 bytes 2024-09-12T16:26:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1498622976 bytes 2024-09-12T16:26:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1574642688 bytes 2024-09-12T16:26:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1637445632 bytes 2024-09-12T16:26:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1663036416 bytes 2024-09-12T16:26:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1697755136 bytes 2024-09-12T16:26:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1738229760 bytes 2024-09-12T16:26:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1788848128 bytes 2024-09-12T16:26:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1846497280 bytes 2024-09-12T16:26:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1857711104 bytes 2024-09-12T16:26:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1913818112 bytes 2024-09-12T16:26:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1930093568 bytes 2024-09-12T16:26:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1972885504 bytes 2024-09-12T16:26:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2012817408 bytes 2024-09-12T16:26:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2040537088 bytes 2024-09-12T16:26:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2086407168 bytes 2024-09-12T16:26:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2123288576 bytes 2024-09-12T16:26:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2222196736 bytes 2024-09-12T16:26:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2286588928 bytes 2024-09-12T16:26:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2310462464 bytes 2024-09-12T16:26:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2332484608 bytes 2024-09-12T16:26:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2420502528 bytes 2024-09-12T16:26:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2426757120 bytes 2024-09-12T16:26:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2434514944 bytes 2024-09-12T16:26:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2470663168 bytes 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:26:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:26:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:26:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T16:26:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_5 2024-09-12T16:26:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T16:26:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b82c10a5 2024-09-12T16:26:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:26:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_5 present 128b82c10a5 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b82c10a5 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:26:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T16:26:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:26:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:26:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_5 2024-09-12T16:26:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T16:26:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:26:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b82c10a5 2024-09-12T16:26:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b82c10a5 2024-09-12T16:26:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b82c10a5 -o 128b82c10a5 = 128b82c10a5 2024-09-12T16:26:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:26:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:26:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:26:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-12T16:26:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:26:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:26:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:26:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-12T16:27:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:27:21.602+0000 7f92f95d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6cff0b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:27:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:27:21.602+0000 7f92ffde5640 -1 rbd::mirror::ImageReplayer: 0x557c6d6da780 [3/f8418aca-7b96-4b9d-8453-91acf73b514b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:27:23.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:27:23.603+0000 7f40b94a4640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b46b5800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2024-09-12T16:27:23.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:27:23.603+0000 7f40b94a4640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b46b5800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2024-09-12T16:27:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:27:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:27:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T16:27:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:27:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:27:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_5 2024-09-12T16:27:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T16:27:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:27:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T16:27:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:27:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:27:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:27:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_5 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_5 2024-09-12T16:27:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-12T16:27:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:27:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-12T16:27:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:27:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:27:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:27:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:27:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-12T16:27:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:27:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:27:23.828 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-12T16:27:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-12T16:27:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:27:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:27:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:27:23.875 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:27:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_5 2024-09-12T16:27:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:27:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_5 2024-09-12T16:27:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-12T16:27:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-12T16:27:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:27:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:27:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:27:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_6 2024-09-12T16:27:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:27:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-12T16:27:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-12T16:27:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:27:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:27:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:27:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-12T16:27:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:27:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-12T16:27:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_6': 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 198ebcbec46f 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.198ebcbec46f 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:27:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:27:24 2024 2024-09-12T16:27:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:27:24 2024 2024-09-12T16:27:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:27:24 2024 2024-09-12T16:27:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:27:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:27:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: cd61b272-088f-4b09-8de6-f83429640dbe 2024-09-12T16:27:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:27:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-12T16:27:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:27:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:27:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:27:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=7 2024-09-12T16:27:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:27:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:27:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 41417728 bytes 2024-09-12T16:27:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 77289472 bytes 2024-09-12T16:27:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 128720896 bytes 2024-09-12T16:27:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 177883136 bytes 2024-09-12T16:27:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 192742400 bytes 2024-09-12T16:27:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 222288896 bytes 2024-09-12T16:27:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 273423360 bytes 2024-09-12T16:27:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 329545728 bytes 2024-09-12T16:27:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 432798720 bytes 2024-09-12T16:27:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 485804032 bytes 2024-09-12T16:27:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 566906880 bytes 2024-09-12T16:27:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 610553856 bytes 2024-09-12T16:27:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 653471744 bytes 2024-09-12T16:27:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 697959424 bytes 2024-09-12T16:27:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 749543424 bytes 2024-09-12T16:27:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 793573376 bytes 2024-09-12T16:27:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 811155456 bytes 2024-09-12T16:27:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 857304064 bytes 2024-09-12T16:27:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 880933888 bytes 2024-09-12T16:27:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 958732288 bytes 2024-09-12T16:27:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1012448256 bytes 2024-09-12T16:27:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1014450176 bytes 2024-09-12T16:27:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1050926080 bytes 2024-09-12T16:27:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1073751040 bytes 2024-09-12T16:27:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1110943744 bytes 2024-09-12T16:27:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1197606912 bytes 2024-09-12T16:27:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1201804288 bytes 2024-09-12T16:27:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1251402752 bytes 2024-09-12T16:27:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1368800256 bytes 2024-09-12T16:27:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1409389568 bytes 2024-09-12T16:27:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1478007808 bytes 2024-09-12T16:27:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1579317248 bytes 2024-09-12T16:27:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1639103488 bytes 2024-09-12T16:27:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1702714368 bytes 2024-09-12T16:27:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1766705152 bytes 2024-09-12T16:27:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1800568832 bytes 2024-09-12T16:27:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:27:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:27:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:27:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:27:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-12T16:27:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_6 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T16:27:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:27:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128bad80f4fa 2024-09-12T16:27:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:27:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:27:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_6 present 128bad80f4fa 2024-09-12T16:27:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:27:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128bad80f4fa 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_6 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T16:27:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128bad80f4fa 2024-09-12T16:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128bad80f4fa 2024-09-12T16:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128bad80f4fa -o 128bad80f4fa = 128bad80f4fa 2024-09-12T16:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:27:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-12T16:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:27:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-12T16:27:51.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:27:51.602+0000 7f92f95d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6d66bd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:27:51.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:27:51.602+0000 7f92ffde5640 -1 rbd::mirror::ImageReplayer: 0x557c6d6daa00 [3/cd61b272-088f-4b09-8de6-f83429640dbe] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:28:00.727 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5363303771801253) > weight (0.25) 2024-09-12T16:28:00.727 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-12T16:28:00.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-12T16:28:00.728 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-12T16:28:00.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:28:00.728+0000 7fc178d8f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 57770) UID: 1000 2024-09-12T16:28:03.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:28:03.613+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b477f400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-12T16:28:03.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:28:03.613+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b477f400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-12T16:28:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:28:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T16:28:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_6 2024-09-12T16:28:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T16:28:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T16:28:03.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:28:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:28:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:28:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_6 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_6 2024-09-12T16:28:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-12T16:28:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-12T16:28:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:28:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-12T16:28:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-12T16:28:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:28:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:28:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:28:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_6 2024-09-12T16:28:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_6 2024-09-12T16:28:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-12T16:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-12T16:28:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_7 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-12T16:28:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-12T16:28:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:28:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-12T16:28:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:28:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_7': 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 19a6323445c2 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.19a6323445c2 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:28:04 2024 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:28:04 2024 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:28:04 2024 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 78b6c648-69c7-48be-9f9d-cad8f139d2d8 2024-09-12T16:28:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:28:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-12T16:28:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:28:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=14 2024-09-12T16:28:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:28:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:28:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 7942144 bytes 2024-09-12T16:28:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 43742208 bytes 2024-09-12T16:28:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 60216320 bytes 2024-09-12T16:28:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 175725568 bytes 2024-09-12T16:28:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 189699072 bytes 2024-09-12T16:28:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 225770496 bytes 2024-09-12T16:28:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 256814080 bytes 2024-09-12T16:28:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 274011136 bytes 2024-09-12T16:28:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 357019648 bytes 2024-09-12T16:28:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 449085440 bytes 2024-09-12T16:28:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 510264320 bytes 2024-09-12T16:28:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 622350336 bytes 2024-09-12T16:28:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 652464128 bytes 2024-09-12T16:28:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 757430272 bytes 2024-09-12T16:28:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 812346368 bytes 2024-09-12T16:28:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 835382272 bytes 2024-09-12T16:28:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 900340736 bytes 2024-09-12T16:28:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 1034015744 bytes 2024-09-12T16:28:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 1046679552 bytes 2024-09-12T16:28:08.729 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-12T16:28:08.729 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:28:08.730 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-12T16:28:08.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-12T16:28:08.730 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-12T16:28:08.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-12T16:28:08.736 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-09-12T16:28:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 1078836224 bytes 2024-09-12T16:28:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 1133694976 bytes 2024-09-12T16:28:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 1230108672 bytes 2024-09-12T16:28:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1269308416 bytes 2024-09-12T16:28:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1283085312 bytes 2024-09-12T16:28:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1339170816 bytes 2024-09-12T16:28:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1397799936 bytes 2024-09-12T16:28:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1427189760 bytes 2024-09-12T16:28:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1471167488 bytes 2024-09-12T16:28:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1503653888 bytes 2024-09-12T16:28:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1582406656 bytes 2024-09-12T16:28:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1641355264 bytes 2024-09-12T16:28:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1739826176 bytes 2024-09-12T16:28:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1740585984 bytes 2024-09-12T16:28:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1760205824 bytes 2024-09-12T16:28:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1774051328 bytes 2024-09-12T16:28:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1810674688 bytes 2024-09-12T16:28:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1884370944 bytes 2024-09-12T16:28:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1964755968 bytes 2024-09-12T16:28:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 2005574656 bytes 2024-09-12T16:28:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 2039557120 bytes 2024-09-12T16:28:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 2111316992 bytes 2024-09-12T16:28:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 2142245888 bytes 2024-09-12T16:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 2184496128 bytes 2024-09-12T16:28:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 2206223360 bytes 2024-09-12T16:28:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 2223195136 bytes 2024-09-12T16:28:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 2243789824 bytes 2024-09-12T16:28:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 2302880768 bytes 2024-09-12T16:28:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 2303504384 bytes 2024-09-12T16:28:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 2315428864 bytes 2024-09-12T16:28:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 2355615744 bytes 2024-09-12T16:28:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2431938560 bytes 2024-09-12T16:28:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2509281280 bytes 2024-09-12T16:28:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2528611328 bytes 2024-09-12T16:28:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5400 IOs, 2599099392 bytes 2024-09-12T16:28:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5500 IOs, 2659300352 bytes 2024-09-12T16:28:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5600 IOs, 2688329728 bytes 2024-09-12T16:28:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5700 IOs, 2721800192 bytes 2024-09-12T16:28:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5800 IOs, 2736392192 bytes 2024-09-12T16:28:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5900 IOs, 2761349120 bytes 2024-09-12T16:28:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6000 IOs, 2780085248 bytes 2024-09-12T16:28:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6100 IOs, 2817938432 bytes 2024-09-12T16:28:16.216 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6200 IOs, 2863059968 bytes 2024-09-12T16:28:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6300 IOs, 2891998208 bytes 2024-09-12T16:28:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6400 IOs, 2977196032 bytes 2024-09-12T16:28:16.839 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6500 IOs, 3005873152 bytes 2024-09-12T16:28:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6600 IOs, 3032802304 bytes 2024-09-12T16:28:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6700 IOs, 3050306560 bytes 2024-09-12T16:28:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6800 IOs, 3140830208 bytes 2024-09-12T16:28:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:6900 IOs, 3168594944 bytes 2024-09-12T16:28:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7000 IOs, 3206961152 bytes 2024-09-12T16:28:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7100 IOs, 3226912768 bytes 2024-09-12T16:28:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7200 IOs, 3290756096 bytes 2024-09-12T16:28:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7300 IOs, 3315666944 bytes 2024-09-12T16:28:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7400 IOs, 3423557632 bytes 2024-09-12T16:28:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:7500 IOs, 3466874880 bytes 2024-09-12T16:28:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:28:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T16:28:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_7 2024-09-12T16:28:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T16:28:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b384120bf 2024-09-12T16:28:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:28:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_7 present 128b384120bf 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b384120bf 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T16:28:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_7 2024-09-12T16:28:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T16:28:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b384120bf 2024-09-12T16:28:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b384120bf 2024-09-12T16:28:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b384120bf -o 128b384120bf = 128b384120bf 2024-09-12T16:28:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:28:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:28:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-12T16:28:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-12T16:28:21.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:28:21.602+0000 7f92f95d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6cff5520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:28:21.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:28:21.602+0000 7f92ffde5640 -1 rbd::mirror::ImageReplayer: 0x557c6d6daa00 [3/78b6c648-69c7-48be-9f9d-cad8f139d2d8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:28:39.455 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5534870087723259) > weight (0.25) 2024-09-12T16:28:39.456 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7000353315794241) > weight (0.25) 2024-09-12T16:28:39.456 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8788552814046499) > weight (0.25) 2024-09-12T16:28:39.456 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.3197702383492066) > weight (0.25) 2024-09-12T16:28:39.456 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-12T16:28:48.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:28:48.463+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b11aa400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-09-12T16:28:48.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:28:48.463+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b11aa400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-09-12T16:28:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:28:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:28:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T16:28:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_7 2024-09-12T16:28:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T16:28:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T16:28:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:28:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:28:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:28:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-12T16:28:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_7 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_7 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-12T16:28:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-12T16:28:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-12T16:28:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-12T16:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:28:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:28:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_7 2024-09-12T16:28:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_7 2024-09-12T16:28:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-12T16:28:48.786 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-12T16:28:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:28:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_8 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:28:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:28:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-12T16:28:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-12T16:28:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:28:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:28:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-12T16:28:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:28:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-12T16:28:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_8': 2024-09-12T16:28:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:28:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:28:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:28:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 19bfd160aba8 2024-09-12T16:28:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.19bfd160aba8 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:28:49 2024 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:28:49 2024 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:28:49 2024 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 0a17e941-d0f0-45a7-ac78-347365cc8028 2024-09-12T16:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-12T16:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:28:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:28:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=6 2024-09-12T16:28:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:28:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:28:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 23845888 bytes 2024-09-12T16:28:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 66280448 bytes 2024-09-12T16:28:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 68484096 bytes 2024-09-12T16:28:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 102266880 bytes 2024-09-12T16:28:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 111279104 bytes 2024-09-12T16:28:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 135853056 bytes 2024-09-12T16:28:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 167872512 bytes 2024-09-12T16:28:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 205401088 bytes 2024-09-12T16:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 213647360 bytes 2024-09-12T16:28:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 236097536 bytes 2024-09-12T16:28:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 280981504 bytes 2024-09-12T16:28:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 281609216 bytes 2024-09-12T16:28:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 318709760 bytes 2024-09-12T16:28:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 370750464 bytes 2024-09-12T16:28:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 397627392 bytes 2024-09-12T16:28:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 443698176 bytes 2024-09-12T16:28:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 549044224 bytes 2024-09-12T16:28:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 615395328 bytes 2024-09-12T16:28:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 664193024 bytes 2024-09-12T16:28:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 721299456 bytes 2024-09-12T16:28:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 756615168 bytes 2024-09-12T16:28:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 791220224 bytes 2024-09-12T16:28:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 882062336 bytes 2024-09-12T16:28:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 934472704 bytes 2024-09-12T16:28:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1011744768 bytes 2024-09-12T16:28:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1046051840 bytes 2024-09-12T16:28:54.281 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:28:54.280+0000 7fbc5e3a3640 -1 bdev(0x55f8188eec00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbc5e3a3640 time 2024-09-12T16:28:54.281013+0000 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: 664: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f81481cdf3] 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:28:54.281+0000 7fbc5e3a3640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbc5e3a3640 time 2024-09-12T16:28:54.281013+0000 2024-09-12T16:28:54.282 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: 664: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f81481cdf3] 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:*** Caught signal (Aborted) ** 2024-09-12T16:28:54.283 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: in thread 7fbc5e3a3640 thread_name:bstore_aio 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbc6b23e6f0] 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fbc6b28b99c] 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: raise() 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: abort() 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f81481ceb4] 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 7: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.284 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 8: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 9: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:2024-09-12T16:28:54.282+0000 7fbc5e3a3640 -1 *** Caught signal (Aborted) ** 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: in thread 7fbc5e3a3640 thread_name:bstore_aio 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbc6b23e6f0] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fbc6b28b99c] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: raise() 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: abort() 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f81481ceb4] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 7: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 8: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.285 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 9: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.286 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-12T16:28:54.286 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.301 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: -342> 2024-09-12T16:28:54.280+0000 7fbc5e3a3640 -1 bdev(0x55f8188eec00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-09-12T16:28:54.302 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: -101> 2024-09-12T16:28:54.281+0000 7fbc5e3a3640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbc5e3a3640 time 2024-09-12T16:28:54.281013+0000 2024-09-12T16:28:54.302 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: 664: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-09-12T16:28:54.302 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.302 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.302 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f81481cdf3] 2024-09-12T16:28:54.302 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.302 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.303 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.303 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.303 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.303 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 0> 2024-09-12T16:28:54.282+0000 7fbc5e3a3640 -1 *** Caught signal (Aborted) ** 2024-09-12T16:28:54.303 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: in thread 7fbc5e3a3640 thread_name:bstore_aio 2024-09-12T16:28:54.304 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.304 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.304 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbc6b23e6f0] 2024-09-12T16:28:54.304 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fbc6b28b99c] 2024-09-12T16:28:54.304 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: raise() 2024-09-12T16:28:54.304 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: abort() 2024-09-12T16:28:54.304 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f81481ceb4] 2024-09-12T16:28:54.305 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.305 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 7: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.305 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 8: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.305 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 9: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.305 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-12T16:28:54.305 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.310 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: -1022> 2024-09-12T16:28:54.280+0000 7fbc5e3a3640 -1 bdev(0x55f8188eec00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-09-12T16:28:54.310 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: -781> 2024-09-12T16:28:54.281+0000 7fbc5e3a3640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fbc5e3a3640 time 2024-09-12T16:28:54.281013+0000 2024-09-12T16:28:54.310 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-4883-gd61b0886/rpm/el9/BUILD/ceph-19.3.0-4883-gd61b0886/src/blk/kernel/KernelDevice.cc: 664: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55f81481cdf3] 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.311 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: -680> 2024-09-12T16:28:54.282+0000 7fbc5e3a3640 -1 *** Caught signal (Aborted) ** 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: in thread 7fbc5e3a3640 thread_name:bstore_aio 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev) 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbc6b23e6f0] 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fbc6b28b99c] 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 3: raise() 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 4: abort() 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x55f81481ceb4] 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x55f815228240] 2024-09-12T16:28:54.312 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 7: ceph-osd(+0xe1b3b1) [0x55f8152283b1] 2024-09-12T16:28:54.313 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 8: /lib64/libc.so.6(+0x89c52) [0x7fbc6b289c52] 2024-09-12T16:28:54.313 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 9: /lib64/libc.so.6(+0x10ec80) [0x7fbc6b30ec80] 2024-09-12T16:28:54.313 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-12T16:28:54.313 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr: 2024-09-12T16:28:54.433 INFO:tasks.ceph.cluster2.osd.0.smithi019.stderr:daemon-helper: command failed with exit status 5 2024-09-12T16:28:54.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1101480960 bytes 2024-09-12T16:28:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1152414720 bytes 2024-09-12T16:28:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1226644480 bytes 2024-09-12T16:28:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1261436928 bytes 2024-09-12T16:28:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1292853248 bytes 2024-09-12T16:28:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1340083200 bytes 2024-09-12T16:28:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1413735424 bytes 2024-09-12T16:28:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1436002304 bytes 2024-09-12T16:28:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1453876224 bytes 2024-09-12T16:28:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:28:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:28:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:28:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-12T16:28:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:28:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:28:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:28:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:28:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:28:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T16:28:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_8 2024-09-12T16:28:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T16:28:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b90527634 2024-09-12T16:28:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:28:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:28:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_8 present 128b90527634 2024-09-12T16:28:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:28:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:28:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:28:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b90527634 2024-09-12T16:28:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:28:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:28:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T16:28:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:28:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:28:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_8 2024-09-12T16:28:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T16:28:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:28:55.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-09-12T16:28:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b90527634 2024-09-12T16:28:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b90527634 2024-09-12T16:28:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b90527634 -o 128b90527634 = 128b90527634 2024-09-12T16:28:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:28:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:28:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:28:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-12T16:28:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:28:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:28:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:28:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-12T16:29:01.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2024-09-12T16:29:06.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2024-09-12T16:29:11.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2024-09-12T16:29:17.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2024-09-12T16:29:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:29:21.603+0000 7f92f95d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6d66b380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:29:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:29:21.603+0000 7f93005e6640 -1 rbd::mirror::ImageReplayer: 0x557c6d6daa00 [3/0a17e941-d0f0-45a7-ac78-347365cc8028] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:29:22.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2024-09-12T16:29:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:29:26.468+0000 7f40b94a4640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b48b9400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=192 2024-09-12T16:29:26.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:29:26.468+0000 7f40b94a4640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b48b9400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=192, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=c0, last_copied_object_number=128, snap_seqs={192=18446744073709551614}] 2024-09-12T16:29:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:29:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-12T16:29:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:29:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:29:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_8 2024-09-12T16:29:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T16:29:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:29:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T16:29:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:29:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:29:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:29:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_8 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_8 2024-09-12T16:29:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-12T16:29:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:29:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:29:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-12T16:29:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-12T16:29:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:29:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:29:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:29:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:29:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_8 2024-09-12T16:29:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:29:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_8 2024-09-12T16:29:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-12T16:29:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-12T16:29:26.905 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:29:26.902+0000 7ff6f8bb6640 -1 reset not still connected to 0x55fa5764e410 2024-09-12T16:29:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:29:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:29:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_9 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:29:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:29:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:29:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:29:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:29:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:29:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-12T16:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-12T16:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:29:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:29:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-12T16:29:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:29:27.502 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-12T16:29:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_9': 2024-09-12T16:29:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 19d0fd95843d 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.19d0fd95843d 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:29:27 2024 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:29:27 2024 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:29:27 2024 2024-09-12T16:29:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:29:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:29:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 4469659a-87c7-4d72-975e-bdff3272494c 2024-09-12T16:29:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:29:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-12T16:29:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:29:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:29:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:29:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=6 2024-09-12T16:29:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:29:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:29:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 80151552 bytes 2024-09-12T16:29:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 136061952 bytes 2024-09-12T16:29:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 174080000 bytes 2024-09-12T16:29:28.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2024-09-12T16:29:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 193272832 bytes 2024-09-12T16:29:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 250080256 bytes 2024-09-12T16:29:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 279661568 bytes 2024-09-12T16:29:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 286525440 bytes 2024-09-12T16:29:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 300069888 bytes 2024-09-12T16:29:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 320053248 bytes 2024-09-12T16:29:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 351211520 bytes 2024-09-12T16:29:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 425319424 bytes 2024-09-12T16:29:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 495007744 bytes 2024-09-12T16:29:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 527991808 bytes 2024-09-12T16:29:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 560543744 bytes 2024-09-12T16:29:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 605104128 bytes 2024-09-12T16:29:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 665011200 bytes 2024-09-12T16:29:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 719182848 bytes 2024-09-12T16:29:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 784020480 bytes 2024-09-12T16:29:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 831247360 bytes 2024-09-12T16:29:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 877777920 bytes 2024-09-12T16:29:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 892418048 bytes 2024-09-12T16:29:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 953059328 bytes 2024-09-12T16:29:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1034671104 bytes 2024-09-12T16:29:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1047189504 bytes 2024-09-12T16:29:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1047806976 bytes 2024-09-12T16:29:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1125683200 bytes 2024-09-12T16:29:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1158219776 bytes 2024-09-12T16:29:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1160476672 bytes 2024-09-12T16:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1219557376 bytes 2024-09-12T16:29:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1243979776 bytes 2024-09-12T16:29:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1318072320 bytes 2024-09-12T16:29:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1318701056 bytes 2024-09-12T16:29:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1323003904 bytes 2024-09-12T16:29:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1349568512 bytes 2024-09-12T16:29:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1394686976 bytes 2024-09-12T16:29:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1405177856 bytes 2024-09-12T16:29:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1428907008 bytes 2024-09-12T16:29:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1482673152 bytes 2024-09-12T16:29:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1484276736 bytes 2024-09-12T16:29:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1535097856 bytes 2024-09-12T16:29:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1585780736 bytes 2024-09-12T16:29:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1664004096 bytes 2024-09-12T16:29:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4300 IOs, 1725532160 bytes 2024-09-12T16:29:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4400 IOs, 1772241920 bytes 2024-09-12T16:29:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4500 IOs, 1779135488 bytes 2024-09-12T16:29:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4600 IOs, 1839723520 bytes 2024-09-12T16:29:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4700 IOs, 1906461696 bytes 2024-09-12T16:29:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4800 IOs, 1940592640 bytes 2024-09-12T16:29:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4900 IOs, 1973271552 bytes 2024-09-12T16:29:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5000 IOs, 1973899264 bytes 2024-09-12T16:29:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5100 IOs, 2014317568 bytes 2024-09-12T16:29:33.347 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5200 IOs, 2073144320 bytes 2024-09-12T16:29:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:5300 IOs, 2116385792 bytes 2024-09-12T16:29:33.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2024-09-12T16:29:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:29:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T16:29:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:29:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:29:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_9 2024-09-12T16:29:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T16:29:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b48d7977b 2024-09-12T16:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_9 present 128b48d7977b 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b48d7977b 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T16:29:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:29:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:29:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_9 2024-09-12T16:29:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T16:29:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b48d7977b 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b48d7977b 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b48d7977b -o 128b48d7977b = 128b48d7977b 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:29:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:29:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-12T16:29:38.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2024-09-12T16:29:41.737 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7025592650216606) > weight (0.25) 2024-09-12T16:29:41.737 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3679553231800218) > weight (0.25) 2024-09-12T16:29:41.737 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-12T16:29:41.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-12T16:29:41.738 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-12T16:29:41.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:29:41.738+0000 7fad38158640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 63481) UID: 1000 2024-09-12T16:29:44.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2024-09-12T16:29:45.739 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-12T16:29:45.739 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:29:45.739 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-12T16:29:45.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-12T16:29:45.740 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-12T16:29:45.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-12T16:29:45.743 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-12T16:29:49.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2024-09-12T16:29:51.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:29:51.603+0000 7f92f95d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6cff09c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:29:51.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:29:51.603+0000 7f93005e6640 -1 rbd::mirror::ImageReplayer: 0x557c6d6da780 [3/4469659a-87c7-4d72-975e-bdff3272494c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:29:55.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2024-09-12T16:29:55.457 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-12T16:29:55.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-12T16:29:55.457 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-12T16:29:55.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:29:55.457+0000 7efd44791640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 72483) UID: 1000 2024-09-12T16:30:00.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2024-09-12T16:30:01.458 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-12T16:30:01.458 INFO:teuthology.orchestra.run:waiting for 600 2024-09-12T16:30:01.458 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-12T16:30:01.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-12T16:30:01.459 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-12T16:30:01.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-12T16:30:01.462 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-12T16:30:03.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:03.630+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b3953800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-12T16:30:03.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:03.630+0000 7f40b8ca3640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b3953800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-12T16:30:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:30:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-12T16:30:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:30:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:30:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T16:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_9 2024-09-12T16:30:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T16:30:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:30:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T16:30:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:30:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:30:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:30:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:30:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_9 2024-09-12T16:30:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_9 2024-09-12T16:30:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-12T16:30:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:30:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-12T16:30:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:30:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-12T16:30:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-12T16:30:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:30:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:30:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_9 2024-09-12T16:30:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_9 2024-09-12T16:30:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-12T16:30:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-12T16:30:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for i in `seq 1 10` 2024-09-12T16:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image=image_10 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local size=128 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -n 512M ']' 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ size=512M 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ shift 2024-09-12T16:30:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-12T16:30:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-12T16:30:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local mode=snapshot 2024-09-12T16:30:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-12T16:30:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring enabled 2024-09-12T16:30:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-12T16:30:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:rbd image 'image_10': 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: size 512 MiB in 128 objects 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: order 22 (4 MiB objects) 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: snapshot_count: 1 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: id: 19f5e1cfb6d2 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: block_name_prefix: rbd_data.19f5e1cfb6d2 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: format: 2 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: op_features: 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: flags: 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: create_timestamp: Thu Sep 12 16:30:04 2024 2024-09-12T16:30:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: access_timestamp: Thu Sep 12 16:30:04 2024 2024-09-12T16:30:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: modify_timestamp: Thu Sep 12 16:30:04 2024 2024-09-12T16:30:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring state: enabled 2024-09-12T16:30:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring mode: snapshot 2024-09-12T16:30:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring global id: 7f63719e-9979-496d-a1cc-b5ff7442a583 2024-09-12T16:30:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout: mirroring primary: true 2024-09-12T16:30:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-12T16:30:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-12T16:30:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local duration=5 2024-09-12T16:30:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-12T16:30:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:100 IOs, 42747904 bytes 2024-09-12T16:30:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:200 IOs, 121435136 bytes 2024-09-12T16:30:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:300 IOs, 165912576 bytes 2024-09-12T16:30:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:400 IOs, 166526976 bytes 2024-09-12T16:30:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:500 IOs, 219240448 bytes 2024-09-12T16:30:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:600 IOs, 271262720 bytes 2024-09-12T16:30:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:700 IOs, 302118912 bytes 2024-09-12T16:30:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:800 IOs, 345234432 bytes 2024-09-12T16:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:900 IOs, 364956672 bytes 2024-09-12T16:30:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1000 IOs, 395460608 bytes 2024-09-12T16:30:05.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2024-09-12T16:30:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1100 IOs, 458120192 bytes 2024-09-12T16:30:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1200 IOs, 569265152 bytes 2024-09-12T16:30:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1300 IOs, 675109888 bytes 2024-09-12T16:30:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1400 IOs, 687296512 bytes 2024-09-12T16:30:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1500 IOs, 707362816 bytes 2024-09-12T16:30:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1600 IOs, 736334848 bytes 2024-09-12T16:30:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1700 IOs, 795006976 bytes 2024-09-12T16:30:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1800 IOs, 862547968 bytes 2024-09-12T16:30:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:1900 IOs, 868569088 bytes 2024-09-12T16:30:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2000 IOs, 941699072 bytes 2024-09-12T16:30:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2100 IOs, 953016320 bytes 2024-09-12T16:30:07.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2200 IOs, 979945472 bytes 2024-09-12T16:30:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2300 IOs, 1050304512 bytes 2024-09-12T16:30:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2400 IOs, 1069548544 bytes 2024-09-12T16:30:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2500 IOs, 1096858624 bytes 2024-09-12T16:30:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2600 IOs, 1183777792 bytes 2024-09-12T16:30:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2700 IOs, 1226504192 bytes 2024-09-12T16:30:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2800 IOs, 1260498944 bytes 2024-09-12T16:30:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:2900 IOs, 1323219968 bytes 2024-09-12T16:30:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3000 IOs, 1362469888 bytes 2024-09-12T16:30:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3100 IOs, 1404214272 bytes 2024-09-12T16:30:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3200 IOs, 1404836864 bytes 2024-09-12T16:30:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3300 IOs, 1526890496 bytes 2024-09-12T16:30:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3400 IOs, 1588692992 bytes 2024-09-12T16:30:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3500 IOs, 1664057344 bytes 2024-09-12T16:30:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3600 IOs, 1718434816 bytes 2024-09-12T16:30:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3700 IOs, 1771771904 bytes 2024-09-12T16:30:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3800 IOs, 1782745088 bytes 2024-09-12T16:30:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:3900 IOs, 1825495040 bytes 2024-09-12T16:30:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4000 IOs, 1853000704 bytes 2024-09-12T16:30:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4100 IOs, 1866596352 bytes 2024-09-12T16:30:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:4200 IOs, 1887993856 bytes 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=124 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 124 -eq 124 ']' 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=present 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_10 2024-09-12T16:30:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T16:30:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:30:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id=128b6169ce9a 2024-09-12T16:30:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:30:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:30:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_10 present 128b6169ce9a 2024-09-12T16:30:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=present 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id=128b6169ce9a 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_10 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T16:30:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:30:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id=128b6169ce9a 2024-09-12T16:30:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 128b6169ce9a 2024-09-12T16:30:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -z 128b6169ce9a -o 128b6169ce9a = 128b6169ce9a 2024-09-12T16:30:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_state=present 2024-09-12T16:30:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test present = present 2024-09-12T16:30:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-12T16:30:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-12T16:30:11.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2024-09-12T16:30:16.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2024-09-12T16:30:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:21.603+0000 7f92f95d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c6d88cea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-12T16:30:21.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:21.603+0000 7f93005e6640 -1 rbd::mirror::ImageReplayer: 0x557c6d6da780 [3/7f63719e-9979-496d-a1cc-b5ff7442a583] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-12T16:30:22.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2024-09-12T16:30:27.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2024-09-12T16:30:33.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2024-09-12T16:30:38.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2024-09-12T16:30:39.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:39.509+0000 7f40b94a4640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b2089c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-12T16:30:39.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:39.509+0000 7f40b94a4640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5629b2089c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=5d012f07-fdca-4a2f-a7fc-86d6406006b8, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-12T16:30:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:Mirroring disabled 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local state=deleted 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local s 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_10 2024-09-12T16:30:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T16:30:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:30:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T16:30:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ image_id= 2024-09-12T16:30:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-12T16:30:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0.1 2024-09-12T16:30:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-12T16:30:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local test_state=deleted 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image_id= 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_state=deleted 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local current_image_id 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ get_image_id cluster1 mirror image_10 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local image=image_10 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-12T16:30:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-12T16:30:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-12T16:30:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ current_image_id= 2024-09-12T16:30:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n '' 2024-09-12T16:30:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test deleted = deleted 2024-09-12T16:30:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-12T16:30:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-12T16:30:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-12T16:30:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-12T16:30:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ remove_image cluster2 mirror image_10 2024-09-12T16:30:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local image=image_10 2024-09-12T16:30:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-12T16:30:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-12T16:30:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-12T16:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-12T16:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ tee -a /tmp/tmp.65UL20DnMX/rbd-mirror.test.log 2024-09-12T16:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ date '+%F %T' 2024-09-12T16:30:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo 2024-09-12 16:30:40 TEST: check if removed 'images'\''' OMAP are removed 2024-09-12T16:30:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:2024-09-12 16:30:40 TEST: check if removed images' OMAP are removed 2024-09-12T16:30:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-12T16:30:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-12T16:30:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local obj_name=rbd_mirroring 2024-09-12T16:30:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local filter=status_global 2024-09-12T16:30:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-12T16:30:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=status_global 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-12T16:30:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c status_global 2024-09-12T16:30:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 = 0 2024-09-12T16:30:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local obj_name=rbd_mirroring 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local filter=image_ 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=image_ 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-12T16:30:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c image_ 2024-09-12T16:30:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 = 0 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local obj_name=rbd_mirror_leader 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local filter=image_map 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=image_map 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-12T16:30:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c image_map 2024-09-12T16:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 = 0 2024-09-12T16:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=1 2024-09-12T16:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 1 -eq 0 ']' 2024-09-12T16:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 1 2024-09-12T16:30:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=image_map 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-12T16:30:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c image_map 2024-09-12T16:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 1 = 0 2024-09-12T16:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=1 2024-09-12T16:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 1 -eq 0 ']' 2024-09-12T16:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 2 2024-09-12T16:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-12T16:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=image_map 2024-09-12T16:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-12T16:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster1 2024-09-12T16:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c image_map 2024-09-12T16:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-12T16:30:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 = 0 2024-09-12T16:30:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T16:30:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T16:30:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-12T16:30:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local obj_name=rbd_mirroring 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local filter=status_global 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-12T16:30:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster2 2024-09-12T16:30:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=status_global 2024-09-12T16:30:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-12T16:30:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster2 2024-09-12T16:30:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-12T16:30:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c status_global 2024-09-12T16:30:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 = 0 2024-09-12T16:30:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T16:30:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local obj_name=rbd_mirroring 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local filter=image_ 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster2 2024-09-12T16:30:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=image_ 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster2 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirroring 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-12T16:30:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c image_ 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 = 0 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local pool=mirror 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local obj_name=rbd_mirror_leader 2024-09-12T16:30:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local filter=image_map 2024-09-12T16:30:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-12T16:30:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ sleep 0 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster2 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local filter=image_map 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local cluster=cluster2 2024-09-12T16:30:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local pool=mirror 2024-09-12T16:30:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ local obj_name=rbd_mirror_leader 2024-09-12T16:30:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-12T16:30:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ grep -c image_map 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test 0 = 0 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ error_code=0 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set -e 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 0 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ cleanup 0 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local error_code=0 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ set +e 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -ne 0 ']' 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -z '' ']' 2024-09-12T16:30:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster instance 2024-09-12T16:30:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T16:30:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-12T16:30:43.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2024-09-12T16:30:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror' does not exist 2024-09-12T16:30:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T16:30:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-12T16:30:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror' does not exist 2024-09-12T16:30:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T16:30:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-12T16:30:44.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.781+0000 7f40bfcb1640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5629b1a0a900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-12T16:30:44.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.782+0000 7f40bfcb1640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5629b1a0b440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-12T16:30:44.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.782+0000 7f40bfcb1640 -1 rbd::mirror::PoolWatcher: 0x5629b1cbd2c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:44.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.782+0000 7f40b8ca3640 -1 rbd::mirror::PoolWatcher: 0x5629b1cbd4a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:44.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.782+0000 7f40bf4b0640 -1 rbd::mirror::PoolWatcher: 0x5629b1cbde00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:44.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.782+0000 7f40b8ca3640 -1 rbd::mirror::PoolWatcher: 0x5629b1cbc000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:44.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.783+0000 7f40c64be640 -1 rbd::mirror::InstanceWatcher: 0x5629b15bd880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4747 object: (2) No such file or directory 2024-09-12T16:30:44.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.783+0000 7f40c64be640 -1 rbd::mirror::InstanceWatcher: 0x5629b17ba000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4747 object: (2) No such file or directory 2024-09-12T16:30:44.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.783+0000 7f40bf4b0640 -1 rbd::mirror::InstanceWatcher: 0x5629b15bd880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:44.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.783+0000 7f40bf4b0640 -1 rbd::mirror::InstanceWatcher: 0x5629b17ba000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:44.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40b8ca3640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b1648480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:44.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40b94a4640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b1648360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:44.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40c5cbd640 -1 rbd::mirror::NamespaceReplayer: 0x5629b1640000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:44.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40c5cbd640 -1 rbd::mirror::NamespaceReplayer: 0x5629b16401c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:44.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40bf4b0640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b1648000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:44.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40bf4b0640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b1648120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:44.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40c5cbd640 -1 rbd::mirror::NamespaceReplayer: 0x5629b1640000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:44.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:44.784+0000 7f40c5cbd640 -1 rbd::mirror::NamespaceReplayer: 0x5629b16401c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.162+0000 7f40b94a4640 -1 rbd::mirror::RemotePoolPoller: 0x5629b1e061a0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-12T16:30:45.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.162+0000 7f40b8ca3640 -1 rbd::mirror::RemotePoolPoller: 0x5629b1e061a0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-12T16:30:45.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.162+0000 7f40b8ca3640 -1 rbd::mirror::PoolReplayer: 0x5629b03fdc00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-12T16:30:45.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.440+0000 7f40bf4b0640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5629b1a0b680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-12T16:30:45.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.440+0000 7f40b94a4640 -1 rbd::mirror::PoolWatcher: 0x5629b1cbcb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.440+0000 7f40bfcb1640 -1 rbd::mirror::PoolWatcher: 0x5629b1ad7860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.441+0000 7f40c5cbd640 -1 rbd::mirror::LeaderWatcher: 0x5629b1dfbb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-12T16:30:45.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.441+0000 7f40c5cbd640 -1 rbd::mirror::LeaderWatcher: 0x5629b1dfbb00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-12T16:30:45.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.441+0000 7f40c64be640 -1 rbd::mirror::LeaderWatcher: 0x5629b1dfbb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:45.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.442+0000 7f40c64be640 -1 rbd::mirror::InstanceWatcher: 0x5629b15bc000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4747 object: (2) No such file or directory 2024-09-12T16:30:45.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.442+0000 7f40bfcb1640 -1 rbd::mirror::InstanceWatcher: 0x5629b15bc000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:45.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.442+0000 7f40b8ca3640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b13786c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.442+0000 7f40c5cbd640 -1 rbd::mirror::NamespaceReplayer: 0x5629b03fbdc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.442+0000 7f40bf4b0640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b1325320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:45.442+0000 7f40c64be640 -1 rbd::mirror::NamespaceReplayer: 0x5629b03fbdc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f93005e6640 -1 rbd::mirror::LeaderWatcher: 0x557c6dca3b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-12T16:30:45.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f95d8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x557c6d2ca000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-12T16:30:45.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f95d8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x557c6df7e240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-12T16:30:45.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f95d8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x557c6d529440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-12T16:30:45.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f9dd9640 -1 rbd::mirror::PoolWatcher: 0x557c6d0605a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f9dd9640 -1 rbd::mirror::PoolWatcher: 0x557c6c1b0960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f95d8640 -1 rbd::mirror::PoolWatcher: 0x557c6d60eb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f2dcb640 -1 rbd::mirror::PoolWatcher: 0x557c6d061680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f2dcb640 -1 rbd::mirror::PoolWatcher: 0x557c6c1b12c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.671+0000 7f92f2dcb640 -1 rbd::mirror::PoolWatcher: 0x557c6d060000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:45.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.672+0000 7f92ffde5640 -1 rbd::mirror::LeaderWatcher: 0x557c6dca3b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-12T16:30:45.675 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:45.672+0000 7f92ffde5640 -1 rbd::mirror::LeaderWatcher: 0x557c6dca3b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-12T16:30:45.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.870+0000 7f9f84522640 -1 rbd::mirror::RemotePoolPoller: 0x55e770fd4340 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-12T16:30:45.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.870+0000 7f9f83d21640 -1 rbd::mirror::RemotePoolPoller: 0x55e770fd4340 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-12T16:30:45.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.870+0000 7f9f83d21640 -1 rbd::mirror::PoolReplayer: 0x55e76f5bdc00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-12T16:30:45.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.871+0000 7f9f9153c640 -1 rbd::mirror::InstanceWatcher: 0x55e7707aaa80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6153 object: (2) No such file or directory 2024-09-12T16:30:45.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.871+0000 7f9f9153c640 -1 rbd::mirror::InstanceWatcher: 0x55e7707aa380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6153 object: (2) No such file or directory 2024-09-12T16:30:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.871+0000 7f9f8ad2f640 -1 rbd::mirror::InstanceWatcher: 0x55e7707aaa80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.871+0000 7f9f8ad2f640 -1 rbd::mirror::InstanceWatcher: 0x55e7707aa380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f83d21640 -1 rbd::mirror::MirrorStatusUpdater 0x55e7708165a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f83d21640 -1 rbd::mirror::MirrorStatusUpdater 0x55e770816360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f90d3b640 -1 rbd::mirror::NamespaceReplayer: 0x55e76f5bae00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f90d3b640 -1 rbd::mirror::NamespaceReplayer: 0x55e76f5ba000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f8a52e640 -1 rbd::mirror::MirrorStatusUpdater 0x55e770816000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f8a52e640 -1 rbd::mirror::MirrorStatusUpdater 0x55e770816120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f9153c640 -1 rbd::mirror::NamespaceReplayer: 0x55e76f5bae00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.872+0000 7f9f9153c640 -1 rbd::mirror::NamespaceReplayer: 0x55e76f5ba000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.954+0000 7f9f9153c640 -1 rbd::mirror::LeaderWatcher: 0x55e7704a1b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:45.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.955+0000 7f9f9153c640 -1 rbd::mirror::InstanceWatcher: 0x55e7707aa000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6153 object: (2) No such file or directory 2024-09-12T16:30:45.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.955+0000 7f9f8ad2f640 -1 rbd::mirror::InstanceWatcher: 0x55e7707aa000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:45.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.955+0000 7f9f84522640 -1 rbd::mirror::MirrorStatusUpdater 0x55e7705666c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.955+0000 7f9f9153c640 -1 rbd::mirror::NamespaceReplayer: 0x55e76f5ba8c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.956+0000 7f9f8a52e640 -1 rbd::mirror::MirrorStatusUpdater 0x55e770507440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.956+0000 7f9f90d3b640 -1 rbd::mirror::NamespaceReplayer: 0x55e76f5ba8c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:45.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.959+0000 7f9f90d3b640 -1 rbd::mirror::LeaderWatcher: 0x55e7709dd800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:45.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.959+0000 7f9f7cd13640 -1 rbd::mirror::InstanceWatcher: 0x55e770ca2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:45.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.961+0000 7f9f7d514640 -1 rbd::mirror::MirrorStatusUpdater 0x55e7709db9e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:45.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:45.961+0000 7f9f9153c640 -1 rbd::mirror::NamespaceReplayer: 0x55e7709981c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror_parent' does not exist 2024-09-12T16:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ CEPH_ARGS= 2024-09-12T16:30:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-12T16:30:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:pool 'mirror_parent' does not exist 2024-09-12T16:30:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-12T16:30:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirrors cluster1 2024-09-12T16:30:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1 2024-09-12T16:30:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 0 3 2024-09-12T16:30:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster1:0 '' 2024-09-12T16:30:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:0 2024-09-12T16:30:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster1:1 '' 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:1 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster1:2 '' 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:2 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster1:3 '' 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster1:3 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirrors cluster2 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:++ seq 0 3 2024-09-12T16:30:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster2:0 '' 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:0 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster2:1 '' 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:1 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster2:2 '' 2024-09-12T16:30:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:2 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ stop_mirror cluster2:3 '' 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local cluster=cluster2:3 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ local sig= 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test -n 1 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ return 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' -z 1 ']' 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ test '' = /tmp/tmp.65UL20DnMX 2024-09-12T16:30:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ rm -Rf /tmp/tmp.65UL20DnMX 2024-09-12T16:30:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ '[' 0 -eq 0 ']' 2024-09-12T16:30:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ echo OK 2024-09-12T16:30:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stdout:OK 2024-09-12T16:30:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi019.stderr:+ exit 0 2024-09-12T16:30:46.697 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-12T16:30:46.698 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-12T16:30:46.776 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-12T16:30:46.777 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-12T16:30:47.114 DEBUG:teuthology.parallel:result is None 2024-09-12T16:30:47.114 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-12T16:30:47.141 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-12T16:30:47.141 DEBUG:teuthology.orchestra.run.smithi019:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-12T16:30:47.202 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-12T16:30:47.202 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-12T16:30:47.212 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-12T16:30:47.212 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2024-09-12T16:30:47.213 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-12T16:30:47.213 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-12T16:30:47.224 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-12T16:30:47.224 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2024-09-12T16:30:47.224 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-12T16:30:47.224 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.234 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-12T16:30:47.234 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.242+0000 7fe0f99dd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 71635) UID: 1000 2024-09-12T16:30:47.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.243+0000 7fe0f89db640 -1 rbd::mirror::LeaderWatcher: 0x563982547800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.243+0000 7fe0f91dc640 -1 rbd::mirror::InstanceWatcher: 0x563982318700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6435 object: (2) No such file or directory 2024-09-12T16:30:47.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.243+0000 7fe0f91dc640 -1 rbd::mirror::InstanceWatcher: 0x563982318380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6435 object: (2) No such file or directory 2024-09-12T16:30:47.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.243+0000 7fe0f89db640 -1 rbd::mirror::InstanceWatcher: 0x563982318a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6437 object: (2) No such file or directory 2024-09-12T16:30:47.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.243+0000 7fe0f29cf640 -1 rbd::mirror::InstanceWatcher: 0x563982318700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.243+0000 7fe0f29cf640 -1 rbd::mirror::InstanceWatcher: 0x563982318380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.243+0000 7fe0e49b3640 -1 rbd::mirror::InstanceWatcher: 0x563982318a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0e51b4640 -1 rbd::mirror::MirrorStatusUpdater 0x5639825819e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0f91dc640 -1 rbd::mirror::NamespaceReplayer: 0x563982390540 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0ec1c2640 -1 rbd::mirror::MirrorStatusUpdater 0x5639823927e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0ec1c2640 -1 rbd::mirror::MirrorStatusUpdater 0x563982392360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0f89db640 -1 rbd::mirror::NamespaceReplayer: 0x563982390000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0f89db640 -1 rbd::mirror::NamespaceReplayer: 0x5639823901c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0f21ce640 -1 rbd::mirror::MirrorStatusUpdater 0x563982392000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0f21ce640 -1 rbd::mirror::MirrorStatusUpdater 0x563982392120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0f89db640 -1 rbd::mirror::NamespaceReplayer: 0x563982390000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.244+0000 7fe0f89db640 -1 rbd::mirror::NamespaceReplayer: 0x5639823901c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.246+0000 7fe0f89db640 -1 rbd::mirror::LeaderWatcher: 0x563982bf3b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.246+0000 7fe0f89db640 -1 rbd::mirror::InstanceWatcher: 0x563982318000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6435 object: (2) No such file or directory 2024-09-12T16:30:47.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.246+0000 7fe0f29cf640 -1 rbd::mirror::InstanceWatcher: 0x563982318000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.247+0000 7fe0eb9c1640 -1 rbd::mirror::MirrorStatusUpdater 0x5639820d26c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.247+0000 7fe0f89db640 -1 rbd::mirror::NamespaceReplayer: 0x563981166000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.247+0000 7fe0f21ce640 -1 rbd::mirror::MirrorStatusUpdater 0x563982083320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.247+0000 7fe0f91dc640 -1 rbd::mirror::NamespaceReplayer: 0x563981166000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-12T16:30:47.260 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.270 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-12T16:30:47.270 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.300+0000 7fc2f9dc7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 65488) UID: 1000 2024-09-12T16:30:47.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f95c6640 -1 rbd::mirror::LeaderWatcher: 0x55dc1d9b1800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f95c6640 -1 rbd::mirror::InstanceWatcher: 0x55dc1d79e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6049 object: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f95c6640 -1 rbd::mirror::InstanceWatcher: 0x55dc1dc8c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6049 object: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f8dc5640 -1 rbd::mirror::InstanceWatcher: 0x55dc1d79e700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6052 object: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f2db9640 -1 rbd::mirror::InstanceWatcher: 0x55dc1d79e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f2db9640 -1 rbd::mirror::InstanceWatcher: 0x55dc1dc8c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2e559e640 -1 rbd::mirror::InstanceWatcher: 0x55dc1d79e700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f25b8640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1d822000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f25b8640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1d822120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f95c6640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1d820000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.301+0000 7fc2f95c6640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1d8201c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.302+0000 7fc2de590640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1d8226c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.302+0000 7fc2f8dc5640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1d820c40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.302+0000 7fc2e4d9d640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1d9ed9e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.302+0000 7fc2f95c6640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1d820c40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.304+0000 7fc2f8dc5640 -1 rbd::mirror::LeaderWatcher: 0x55dc1e04db00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.304+0000 7fc2f95c6640 -1 rbd::mirror::InstanceWatcher: 0x55dc1d79e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6049 object: (2) No such file or directory 2024-09-12T16:30:47.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.305+0000 7fc2f2db9640 -1 rbd::mirror::InstanceWatcher: 0x55dc1d79e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.305+0000 7fc2f25b8640 -1 rbd::mirror::MirrorStatusUpdater 0x55dc1d521560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.306 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.305+0000 7fc2f95c6640 -1 rbd::mirror::NamespaceReplayer: 0x55dc1c5bc000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-12T16:30:47.317 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.327 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-12T16:30:47.327 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.357+0000 7f9300de7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 67082) UID: 1000 2024-09-12T16:30:47.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92ffde5640 -1 rbd::mirror::InstanceWatcher: 0x557c6d0e6a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6275 object: (2) No such file or directory 2024-09-12T16:30:47.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92ffde5640 -1 rbd::mirror::InstanceWatcher: 0x557c6d0e6380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6275 object: (2) No such file or directory 2024-09-12T16:30:47.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92e5db1640 -1 rbd::mirror::PoolWatcher: 0x557c6d061e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:47.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92f9dd9640 -1 rbd::mirror::InstanceWatcher: 0x557c6d0e6a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92f9dd9640 -1 rbd::mirror::InstanceWatcher: 0x557c6d0e6380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92ffde5640 -1 rbd::mirror::LeaderWatcher: 0x557c6d2b8c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-12T16:30:47.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92ffde5640 -1 rbd::mirror::LeaderWatcher: 0x557c6d2b8c00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-12T16:30:47.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92f35cc640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6d1527e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f92f35cc640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6d152360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f93005e6640 -1 rbd::mirror::NamespaceReplayer: 0x557c6c1e6000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f93005e6640 -1 rbd::mirror::LeaderWatcher: 0x557c6d2b8c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.358+0000 7f93005e6640 -1 rbd::mirror::NamespaceReplayer: 0x557c6c1e7a40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92f95d8640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6d152000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92f95d8640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6d152120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92ffde5640 -1 rbd::mirror::NamespaceReplayer: 0x557c6c1e6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92ffde5640 -1 rbd::mirror::NamespaceReplayer: 0x557c6c1e7a40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92ffde5640 -1 rbd::mirror::InstanceWatcher: 0x557c6d5ca000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6277 object: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92ec5be640 -1 rbd::mirror::InstanceWatcher: 0x557c6d5ca000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92e55b0640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6d33a7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.359+0000 7f92ffde5640 -1 rbd::mirror::NamespaceReplayer: 0x557c6d2c21c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.360+0000 7f92ebdbd640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6d3039e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.360+0000 7f93005e6640 -1 rbd::mirror::NamespaceReplayer: 0x557c6d2c21c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.362+0000 7f92ffde5640 -1 rbd::mirror::LeaderWatcher: 0x557c6dca3b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.362+0000 7f92ffde5640 -1 rbd::mirror::InstanceWatcher: 0x557c6d0e6000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6275 object: (2) No such file or directory 2024-09-12T16:30:47.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.362+0000 7f92f9dd9640 -1 rbd::mirror::InstanceWatcher: 0x557c6d0e6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.363+0000 7f92f2dcb640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6dd646c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.363+0000 7f92ffde5640 -1 rbd::mirror::NamespaceReplayer: 0x557c6c1e6e00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.363+0000 7f92f95d8640 -1 rbd::mirror::MirrorStatusUpdater 0x557c6dd03440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.363+0000 7f93005e6640 -1 rbd::mirror::NamespaceReplayer: 0x557c6c1e6e00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-12T16:30:47.376 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.386 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-12T16:30:47.386 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.416+0000 7f86c5151640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 74325) UID: 1000 2024-09-12T16:30:47.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86c414f640 -1 rbd::mirror::LeaderWatcher: 0x55fb57981b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86c414f640 -1 rbd::mirror::InstanceWatcher: 0x55fb5863a380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6630 object: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86c414f640 -1 rbd::mirror::InstanceWatcher: 0x55fb57c9e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6630 object: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86c4950640 -1 rbd::mirror::InstanceWatcher: 0x55fb5863bc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6633 object: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86bd942640 -1 rbd::mirror::InstanceWatcher: 0x55fb5863a380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86bd942640 -1 rbd::mirror::InstanceWatcher: 0x55fb57c9e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86b0928640 -1 rbd::mirror::InstanceWatcher: 0x55fb5863bc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86b0127640 -1 rbd::mirror::MirrorStatusUpdater 0x55fb579c3b00 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.417+0000 7f86c414f640 -1 rbd::mirror::NamespaceReplayer: 0x55fb577d8fc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86b7135640 -1 rbd::mirror::MirrorStatusUpdater 0x55fb577da7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86c4950640 -1 rbd::mirror::NamespaceReplayer: 0x55fb577d8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86b7936640 -1 rbd::mirror::MirrorStatusUpdater 0x55fb577da360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86c414f640 -1 rbd::mirror::NamespaceReplayer: 0x55fb577d81c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86be143640 -1 rbd::mirror::MirrorStatusUpdater 0x55fb577da000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86c4950640 -1 rbd::mirror::NamespaceReplayer: 0x55fb577d8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86bd942640 -1 rbd::mirror::MirrorStatusUpdater 0x55fb577da120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.418+0000 7f86c4950640 -1 rbd::mirror::NamespaceReplayer: 0x55fb577d81c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.419+0000 7f86c4950640 -1 rbd::mirror::LeaderWatcher: 0x55fb5834db00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.420+0000 7f86c4950640 -1 rbd::mirror::InstanceWatcher: 0x55fb5863a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6630 object: (2) No such file or directory 2024-09-12T16:30:47.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.420+0000 7f86be143640 -1 rbd::mirror::InstanceWatcher: 0x55fb5863a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.421+0000 7f86b7135640 -1 rbd::mirror::MirrorStatusUpdater 0x55fb584106c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.421+0000 7f86c4950640 -1 rbd::mirror::NamespaceReplayer: 0x55fb56897c00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.421+0000 7f86bd942640 -1 rbd::mirror::MirrorStatusUpdater 0x55fb583b3440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.421+0000 7f86c414f640 -1 rbd::mirror::NamespaceReplayer: 0x55fb56897c00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-12T16:30:47.433 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.444 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-12T16:30:47.444 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.474+0000 7f40c6cbf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 47313) UID: 1000 2024-09-12T16:30:47.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.475+0000 7f40b2496640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5629b158e240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-12T16:30:47.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.475+0000 7f40b1c95640 -1 rbd::mirror::PoolWatcher: 0x5629b03c4000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:47.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.475+0000 7f40abc89640 -1 rbd::mirror::PoolWatcher: 0x5629b1ad63c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-12T16:30:47.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.476+0000 7f40c64be640 -1 rbd::mirror::LeaderWatcher: 0x5629b17fd800 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-12T16:30:47.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.476+0000 7f40c64be640 -1 rbd::mirror::LeaderWatcher: 0x5629b17fd800 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-12T16:30:47.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.476+0000 7f40c5cbd640 -1 rbd::mirror::LeaderWatcher: 0x5629b17fd800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.477+0000 7f40c5cbd640 -1 rbd::mirror::InstanceWatcher: 0x5629b17ba380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4750 object: (2) No such file or directory 2024-09-12T16:30:47.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.477+0000 7f40b2496640 -1 rbd::mirror::InstanceWatcher: 0x5629b17ba380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.478+0000 7f40ab488640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b1648900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.478+0000 7f40c64be640 -1 rbd::mirror::NamespaceReplayer: 0x5629b1640fc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.478+0000 7f40b1c95640 -1 rbd::mirror::MirrorStatusUpdater 0x5629b17ffc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi019.stderr:2024-09-12T16:30:47.478+0000 7f40c5cbd640 -1 rbd::mirror::NamespaceReplayer: 0x5629b1640fc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-12T16:30:47.492 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.503 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-12T16:30:47.503 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi019.stderr:2024-09-12T16:30:47.532+0000 7f9f91d3d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 74219) UID: 1000 2024-09-12T16:30:47.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-12T16:30:47.542 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.553 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-12T16:30:47.553 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.583+0000 7fbc38e49640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 73487) UID: 1000 2024-09-12T16:30:47.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.583+0000 7fbc37e47640 -1 rbd::mirror::LeaderWatcher: 0x5611f3d4d800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc3163a640 -1 rbd::mirror::InstanceWatcher: 0x5611f3b21880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc31e3b640 -1 rbd::mirror::InstanceWatcher: 0x5611f3b21c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc2ae2d640 -1 rbd::mirror::MirrorStatusUpdater 0x5611f3b9a5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc37e47640 -1 rbd::mirror::InstanceWatcher: 0x5611f402e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6116 object: (2) No such file or directory 2024-09-12T16:30:47.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc2ae2d640 -1 rbd::mirror::MirrorStatusUpdater 0x5611f3b9a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc37e47640 -1 rbd::mirror::NamespaceReplayer: 0x5611f3b98000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc37e47640 -1 rbd::mirror::NamespaceReplayer: 0x5611f3b981c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.584+0000 7fbc24620640 -1 rbd::mirror::InstanceWatcher: 0x5611f402e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.585+0000 7fbc1d612640 -1 rbd::mirror::MirrorStatusUpdater 0x5611f3b9a6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.585+0000 7fbc37e47640 -1 rbd::mirror::NamespaceReplayer: 0x5611f3b98a80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.585+0000 7fbc23e1f640 -1 rbd::mirror::MirrorStatusUpdater 0x5611f3d6bc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.585+0000 7fbc38648640 -1 rbd::mirror::NamespaceReplayer: 0x5611f3b98a80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.587+0000 7fbc38648640 -1 rbd::mirror::LeaderWatcher: 0x5611f43edb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.588+0000 7fbc31e3b640 -1 rbd::mirror::InstanceWatcher: 0x5611f3b20000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.588+0000 7fbc2b62e640 -1 rbd::mirror::MirrorStatusUpdater 0x5611f38dc6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.588+0000 7fbc38648640 -1 rbd::mirror::NamespaceReplayer: 0x5611f294fa40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.588+0000 7fbc3163a640 -1 rbd::mirror::MirrorStatusUpdater 0x5611f3869320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi019.stderr:2024-09-12T16:30:47.588+0000 7fbc37e47640 -1 rbd::mirror::NamespaceReplayer: 0x5611f294fa40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-12T16:30:47.600 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-12T16:30:47.611 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-12T16:30:47.611 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:30:47.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.640+0000 7f41f9551640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72377) UID: 1000 2024-09-12T16:30:47.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.641+0000 7f41f8d50640 -1 rbd::mirror::LeaderWatcher: 0x559804a5d800 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.641+0000 7f41f854f640 -1 rbd::mirror::InstanceWatcher: 0x559804840380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6069 object: (2) No such file or directory 2024-09-12T16:30:47.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.641+0000 7f41f854f640 -1 rbd::mirror::InstanceWatcher: 0x559804840a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6069 object: (2) No such file or directory 2024-09-12T16:30:47.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.641+0000 7f41f854f640 -1 rbd::mirror::InstanceWatcher: 0x559804cfe000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6071 object: (2) No such file or directory 2024-09-12T16:30:47.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.641+0000 7f41f1d42640 -1 rbd::mirror::InstanceWatcher: 0x559804840380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.641+0000 7f41f1d42640 -1 rbd::mirror::InstanceWatcher: 0x559804840a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41e4527640 -1 rbd::mirror::InstanceWatcher: 0x559804cfe000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41ebd36640 -1 rbd::mirror::MirrorStatusUpdater 0x5598048bc360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41ebd36640 -1 rbd::mirror::MirrorStatusUpdater 0x5598048bc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f854f640 -1 rbd::mirror::NamespaceReplayer: 0x5598048ba000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f854f640 -1 rbd::mirror::NamespaceReplayer: 0x5598048ba1c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f2543640 -1 rbd::mirror::MirrorStatusUpdater 0x5598048bc000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f2543640 -1 rbd::mirror::MirrorStatusUpdater 0x5598048bc120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f8d50640 -1 rbd::mirror::NamespaceReplayer: 0x5598048ba000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f8d50640 -1 rbd::mirror::NamespaceReplayer: 0x5598048ba1c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41de51b640 -1 rbd::mirror::MirrorStatusUpdater 0x559804a947e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f854f640 -1 rbd::mirror::NamespaceReplayer: 0x5598048bac40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41e4d28640 -1 rbd::mirror::MirrorStatusUpdater 0x559804a658c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.642+0000 7f41f8d50640 -1 rbd::mirror::NamespaceReplayer: 0x5598048bac40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.644+0000 7f41f854f640 -1 rbd::mirror::LeaderWatcher: 0x559805125b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-12T16:30:47.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.644+0000 7f41f8d50640 -1 rbd::mirror::InstanceWatcher: 0x559804840000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6069 object: (2) No such file or directory 2024-09-12T16:30:47.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.645+0000 7f41f1d42640 -1 rbd::mirror::InstanceWatcher: 0x559804840000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-12T16:30:47.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.645+0000 7f41eb535640 -1 rbd::mirror::MirrorStatusUpdater 0x5598045fa6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.645+0000 7f41f854f640 -1 rbd::mirror::NamespaceReplayer: 0x5598036a1a40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.645+0000 7f41f2543640 -1 rbd::mirror::MirrorStatusUpdater 0x559804599320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-12T16:30:47.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi019.stderr:2024-09-12T16:30:47.645+0000 7f41f8d50640 -1 rbd::mirror::NamespaceReplayer: 0x5598036a1a40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-12T16:30:47.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-12T16:30:47.658 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-12T16:30:47.669 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-12T16:30:47.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:47.994 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:47.994 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:48.007 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2024-09-12T16:30:47.421351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432504,"kb_used_data":38424,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279683016,"statfs":{"total":289910292480,"available":286395408384,"internally_reserved":0,"allocated":39346176,"data_stored":30117517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536821998,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-140644,"num_read_kb":-39522729,"num_write":-436728,"num_write_kb":-158663501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-5,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.932007"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549967+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:44.549967+0000","last_peered":"2024-09-12T16:30:44.549967+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:44.549967+0000","last_undegraded":"2024-09-12T16:30:44.549967+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.550167+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:44.550167+0000","last_peered":"2024-09-12T16:30:44.550167+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:44.550167+0000","last_undegraded":"2024-09-12T16:30:44.550167+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549372+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:44.549372+0000","last_peered":"2024-09-12T16:30:44.549372+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:44.549372+0000","last_undegraded":"2024-09-12T16:30:44.549372+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.548846+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:44.548846+0000","last_peered":"2024-09-12T16:30:44.548846+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:44.548846+0000","last_undegraded":"2024-09-12T16:30:44.548846+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":46,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.551132+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:44.551132+0000","last_peered":"2024-09-12T16:30:44.551132+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:44.551132+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.764295+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:44.764295+0000","last_peered":"2024-09-12T16:30:44.764295+0000","last_clean":"2024-09-12T16:30:44.764295+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:44.764295+0000","last_undegraded":"2024-09-12T16:30:44.764295+0000","last_fullsized":"2024-09-12T16:30:44.764295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.762233+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:44.762233+0000","last_peered":"2024-09-12T16:30:44.762233+0000","last_clean":"2024-09-12T16:30:44.762233+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:44.762233+0000","last_undegraded":"2024-09-12T16:30:44.762233+0000","last_fullsized":"2024-09-12T16:30:44.762233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1018,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.552114+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:44.552114+0000","last_peered":"2024-09-12T16:30:44.552114+0000","last_clean":"2024-09-12T16:30:44.552114+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:44.552114+0000","last_undegraded":"2024-09-12T16:30:44.552114+0000","last_fullsized":"2024-09-12T16:30:44.552114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1088,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.544684+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:44.544684+0000","last_peered":"2024-09-12T16:30:44.544684+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:44.544684+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673398,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119000,"kb_used_data":6468,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252840,"statfs":{"total":96636764160,"available":95490908160,"internally_reserved":0,"allocated":6623232,"data_stored":3590561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933648,"kb_used_data":5884,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438192,"statfs":{"total":96636764160,"available":95680708608,"internally_reserved":0,"allocated":6025216,"data_stored":3000174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:48.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:48.301 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:48.301 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:48.313 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2024-09-12T16:30:47.421351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432504,"kb_used_data":38424,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279683016,"statfs":{"total":289910292480,"available":286395408384,"internally_reserved":0,"allocated":39346176,"data_stored":30117517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536821998,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-140644,"num_read_kb":-39522729,"num_write":-436728,"num_write_kb":-158663501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-5,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.932007"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549967+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:44.549967+0000","last_peered":"2024-09-12T16:30:44.549967+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:44.549967+0000","last_undegraded":"2024-09-12T16:30:44.549967+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.550167+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:44.550167+0000","last_peered":"2024-09-12T16:30:44.550167+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:44.550167+0000","last_undegraded":"2024-09-12T16:30:44.550167+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549372+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:44.549372+0000","last_peered":"2024-09-12T16:30:44.549372+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:44.549372+0000","last_undegraded":"2024-09-12T16:30:44.549372+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.548846+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:44.548846+0000","last_peered":"2024-09-12T16:30:44.548846+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:44.548846+0000","last_undegraded":"2024-09-12T16:30:44.548846+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":46,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.551132+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:44.551132+0000","last_peered":"2024-09-12T16:30:44.551132+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:44.551132+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.764295+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:44.764295+0000","last_peered":"2024-09-12T16:30:44.764295+0000","last_clean":"2024-09-12T16:30:44.764295+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:44.764295+0000","last_undegraded":"2024-09-12T16:30:44.764295+0000","last_fullsized":"2024-09-12T16:30:44.764295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.762233+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:44.762233+0000","last_peered":"2024-09-12T16:30:44.762233+0000","last_clean":"2024-09-12T16:30:44.762233+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:44.762233+0000","last_undegraded":"2024-09-12T16:30:44.762233+0000","last_fullsized":"2024-09-12T16:30:44.762233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1018,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.552114+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:44.552114+0000","last_peered":"2024-09-12T16:30:44.552114+0000","last_clean":"2024-09-12T16:30:44.552114+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:44.552114+0000","last_undegraded":"2024-09-12T16:30:44.552114+0000","last_fullsized":"2024-09-12T16:30:44.552114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1088,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.544684+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:44.544684+0000","last_peered":"2024-09-12T16:30:44.544684+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:44.544684+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673398,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119000,"kb_used_data":6468,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252840,"statfs":{"total":96636764160,"available":95490908160,"internally_reserved":0,"allocated":6623232,"data_stored":3590561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933648,"kb_used_data":5884,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438192,"statfs":{"total":96636764160,"available":95680708608,"internally_reserved":0,"allocated":6025216,"data_stored":3000174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:48.314 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:48.610 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:48.611 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:48.623 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2024-09-12T16:30:47.421351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432504,"kb_used_data":38424,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279683016,"statfs":{"total":289910292480,"available":286395408384,"internally_reserved":0,"allocated":39346176,"data_stored":30117517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536821998,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-140644,"num_read_kb":-39522729,"num_write":-436728,"num_write_kb":-158663501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-5,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.932007"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549967+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:44.549967+0000","last_peered":"2024-09-12T16:30:44.549967+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:44.549967+0000","last_undegraded":"2024-09-12T16:30:44.549967+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.550167+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:44.550167+0000","last_peered":"2024-09-12T16:30:44.550167+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:44.550167+0000","last_undegraded":"2024-09-12T16:30:44.550167+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549372+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:44.549372+0000","last_peered":"2024-09-12T16:30:44.549372+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:44.549372+0000","last_undegraded":"2024-09-12T16:30:44.549372+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.548846+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:44.548846+0000","last_peered":"2024-09-12T16:30:44.548846+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:44.548846+0000","last_undegraded":"2024-09-12T16:30:44.548846+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":46,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.551132+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:44.551132+0000","last_peered":"2024-09-12T16:30:44.551132+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:44.551132+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.764295+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:44.764295+0000","last_peered":"2024-09-12T16:30:44.764295+0000","last_clean":"2024-09-12T16:30:44.764295+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:44.764295+0000","last_undegraded":"2024-09-12T16:30:44.764295+0000","last_fullsized":"2024-09-12T16:30:44.764295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.762233+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:44.762233+0000","last_peered":"2024-09-12T16:30:44.762233+0000","last_clean":"2024-09-12T16:30:44.762233+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:44.762233+0000","last_undegraded":"2024-09-12T16:30:44.762233+0000","last_fullsized":"2024-09-12T16:30:44.762233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1018,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.552114+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:44.552114+0000","last_peered":"2024-09-12T16:30:44.552114+0000","last_clean":"2024-09-12T16:30:44.552114+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:44.552114+0000","last_undegraded":"2024-09-12T16:30:44.552114+0000","last_fullsized":"2024-09-12T16:30:44.552114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1088,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.544684+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:44.544684+0000","last_peered":"2024-09-12T16:30:44.544684+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:44.544684+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673398,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119000,"kb_used_data":6468,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252840,"statfs":{"total":96636764160,"available":95490908160,"internally_reserved":0,"allocated":6623232,"data_stored":3590561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933648,"kb_used_data":5884,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438192,"statfs":{"total":96636764160,"available":95680708608,"internally_reserved":0,"allocated":6025216,"data_stored":3000174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:48.624 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:30:48.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1019, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:44.549967+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:44.549967+0000', 'last_peered': '2024-09-12T16:30:44.549967+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:44.549967+0000', 'last_undegraded': '2024-09-12T16:30:44.549967+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:48.624 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:30:48.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1019, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:44.550167+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:44.550167+0000', 'last_peered': '2024-09-12T16:30:44.550167+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:44.550167+0000', 'last_undegraded': '2024-09-12T16:30:44.550167+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:48.624 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:30:48.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1020, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:44.549372+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:44.549372+0000', 'last_peered': '2024-09-12T16:30:44.549372+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:44.549372+0000', 'last_undegraded': '2024-09-12T16:30:44.549372+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:48.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:30:48.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1020, 'reported_epoch': 508, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:44.548846+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:44.548846+0000', 'last_peered': '2024-09-12T16:30:44.548846+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:44.548846+0000', 'last_undegraded': '2024-09-12T16:30:44.548846+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:48.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:30:48.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 46, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:44.551132+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:44.551132+0000', 'last_peered': '2024-09-12T16:30:44.551132+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:44.551132+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:48.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:30:48.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1088, 'reported_epoch': 508, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:44.544684+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:44.544684+0000', 'last_peered': '2024-09-12T16:30:44.544684+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:44.544684+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:48.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:30:48.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:48.999 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2169,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514884096,"bytes_avail":286395408384,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":49,"modified":"2024-09-12T16:30:47.421404+0000","services":{"rbd-mirror":{"daemons":{"summary":"","6627":{"start_epoch":48,"start_stamp":"2024-09-12T16:30:01.549999+0000","gid":6627,"addr":"172.21.15.19:0/972492461","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-4883-gd61b0886 (d61b08865d4d98738f45ff2e4806d7fb6d03eca1) squid (dev)","ceph_version_short":"19.3.0-4883-gd61b0886","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi019","id":"mirror.0","instance_id":"6627","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024","kernel_version":"5.14.0-505.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32493484","os":"Linux"},"task_status":{}}}}}},"progress_events":{}} 2024-09-12T16:30:49.012 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:30:49.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:49.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2024-09-12T16:30:49.306 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:49.307 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:49.319 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2024-09-12T16:30:47.421351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432504,"kb_used_data":38424,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279683016,"statfs":{"total":289910292480,"available":286395408384,"internally_reserved":0,"allocated":39346176,"data_stored":30117517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536821998,"num_objects":-167,"num_object_clones":0,"num_object_copies":-334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-167,"num_whiteouts":0,"num_read":-140644,"num_read_kb":-39522729,"num_write":-436728,"num_write_kb":-158663501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-5,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.932007"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549967+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:44.549967+0000","last_peered":"2024-09-12T16:30:44.549967+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:44.549967+0000","last_undegraded":"2024-09-12T16:30:44.549967+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1019,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.550167+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:44.550167+0000","last_peered":"2024-09-12T16:30:44.550167+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:44.550167+0000","last_undegraded":"2024-09-12T16:30:44.550167+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.549372+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:44.549372+0000","last_peered":"2024-09-12T16:30:44.549372+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:44.549372+0000","last_undegraded":"2024-09-12T16:30:44.549372+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1020,"reported_epoch":508,"state":"active+undersized","last_fresh":"2024-09-12T16:30:44.548846+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:44.548846+0000","last_peered":"2024-09-12T16:30:44.548846+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:44.548846+0000","last_undegraded":"2024-09-12T16:30:44.548846+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":46,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.551132+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:44.551132+0000","last_peered":"2024-09-12T16:30:44.551132+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:44.551132+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.764295+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:44.764295+0000","last_peered":"2024-09-12T16:30:44.764295+0000","last_clean":"2024-09-12T16:30:44.764295+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:44.764295+0000","last_undegraded":"2024-09-12T16:30:44.764295+0000","last_fullsized":"2024-09-12T16:30:44.764295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.762233+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:44.762233+0000","last_peered":"2024-09-12T16:30:44.762233+0000","last_clean":"2024-09-12T16:30:44.762233+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:44.762233+0000","last_undegraded":"2024-09-12T16:30:44.762233+0000","last_fullsized":"2024-09-12T16:30:44.762233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1018,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.552114+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:44.552114+0000","last_peered":"2024-09-12T16:30:44.552114+0000","last_clean":"2024-09-12T16:30:44.552114+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:44.552114+0000","last_undegraded":"2024-09-12T16:30:44.552114+0000","last_fullsized":"2024-09-12T16:30:44.552114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1088,"reported_epoch":508,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:44.544684+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:44.544684+0000","last_peered":"2024-09-12T16:30:44.544684+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:44.544684+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673398,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119000,"kb_used_data":6468,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252840,"statfs":{"total":96636764160,"available":95490908160,"internally_reserved":0,"allocated":6623232,"data_stored":3590561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933648,"kb_used_data":5884,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438192,"statfs":{"total":96636764160,"available":95680708608,"internally_reserved":0,"allocated":6025216,"data_stored":3000174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:52.321 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:52.629 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:52.630 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:52.642 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-12T16:30:51.422572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432516,"kb_used_data":38436,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279683004,"statfs":{"total":289910292480,"available":286395396096,"internally_reserved":0,"allocated":39358464,"data_stored":30124166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-140907,"num_read_kb":-39522963,"num_write":-436905,"num_write_kb":-158663516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-5,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002650"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":48,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.559958+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.559958+0000","last_peered":"2024-09-12T16:30:46.559958+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.559958+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.764295+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:44.764295+0000","last_peered":"2024-09-12T16:30:44.764295+0000","last_clean":"2024-09-12T16:30:44.764295+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:44.764295+0000","last_undegraded":"2024-09-12T16:30:44.764295+0000","last_fullsized":"2024-09-12T16:30:44.764295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.762233+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:44.762233+0000","last_peered":"2024-09-12T16:30:44.762233+0000","last_clean":"2024-09-12T16:30:44.762233+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:44.762233+0000","last_undegraded":"2024-09-12T16:30:44.762233+0000","last_fullsized":"2024-09-12T16:30:44.762233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1090,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555741+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555741+0000","last_peered":"2024-09-12T16:30:46.555741+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555741+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673399,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933648,"kb_used_data":5884,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438192,"statfs":{"total":96636764160,"available":95680708608,"internally_reserved":0,"allocated":6025216,"data_stored":3000174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:52.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:52.937 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:52.937 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:52.950 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-12T16:30:51.422572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432516,"kb_used_data":38436,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279683004,"statfs":{"total":289910292480,"available":286395396096,"internally_reserved":0,"allocated":39358464,"data_stored":30124166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-140907,"num_read_kb":-39522963,"num_write":-436905,"num_write_kb":-158663516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":-5,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002650"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":48,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.559958+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.559958+0000","last_peered":"2024-09-12T16:30:46.559958+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.559958+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.764295+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:44.764295+0000","last_peered":"2024-09-12T16:30:44.764295+0000","last_clean":"2024-09-12T16:30:44.764295+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:44.764295+0000","last_undegraded":"2024-09-12T16:30:44.764295+0000","last_fullsized":"2024-09-12T16:30:44.764295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1016,"reported_epoch":508,"state":"active+clean","last_fresh":"2024-09-12T16:30:44.762233+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:44.762233+0000","last_peered":"2024-09-12T16:30:44.762233+0000","last_clean":"2024-09-12T16:30:44.762233+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:44.762233+0000","last_undegraded":"2024-09-12T16:30:44.762233+0000","last_fullsized":"2024-09-12T16:30:44.762233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1090,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555741+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555741+0000","last_peered":"2024-09-12T16:30:46.555741+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555741+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673399,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933648,"kb_used_data":5884,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438192,"statfs":{"total":96636764160,"available":95680708608,"internally_reserved":0,"allocated":6025216,"data_stored":3000174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:30:52.951 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:52.952 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:30:52.952 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 48, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.559958+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.559958+0000', 'last_peered': '2024-09-12T16:30:46.559958+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.559958+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:52.952 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:30:52.952 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1090, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555741+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555741+0000', 'last_peered': '2024-09-12T16:30:46.555741+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555741+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:52.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:30:53.321 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:53.321 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2174,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514896384,"bytes_avail":286395396096,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:30:53.333 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:30:53.333 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:53.627 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:53.640 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2024-09-12T16:30:53.423015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2084,"num_read_kb":-1966,"num_write":-410,"num_write_kb":-305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.879655"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":48,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.559958+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.559958+0000","last_peered":"2024-09-12T16:30:46.559958+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.559958+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1090,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555741+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555741+0000","last_peered":"2024-09-12T16:30:46.555741+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555741+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673399,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:54.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2024-09-12T16:30:56.642 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:56.949 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:56.950 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:56.961 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2024-09-12T16:30:55.423400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2084,"num_read_kb":-1966,"num_write":-410,"num_write_kb":-305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":48,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.559958+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.559958+0000","last_peered":"2024-09-12T16:30:46.559958+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.559958+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1090,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555741+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555741+0000","last_peered":"2024-09-12T16:30:46.555741+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555741+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673399,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:56.962 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:57.255 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:57.255 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:57.267 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2024-09-12T16:30:55.423400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2084,"num_read_kb":-1966,"num_write":-410,"num_write_kb":-305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":48,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.559958+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.559958+0000","last_peered":"2024-09-12T16:30:46.559958+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.559958+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1090,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555741+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555741+0000","last_peered":"2024-09-12T16:30:46.555741+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555741+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673399,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:30:57.268 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:30:57.268 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 48, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.559958+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.559958+0000', 'last_peered': '2024-09-12T16:30:46.559958+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.559958+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:30:57.269 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1090, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555741+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555741+0000', 'last_peered': '2024-09-12T16:30:46.555741+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555741+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:30:57.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:30:57.642 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:57.643 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2178,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:30:57.656 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:30:57.656 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:30:57.951 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:30:57.951 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:30:57.963 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1592,"stamp":"2024-09-12T16:30:57.423778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.870090"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":48,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.559958+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.559958+0000","last_peered":"2024-09-12T16:30:46.559958+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.559958+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1090,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555741+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555741+0000","last_peered":"2024-09-12T16:30:46.555741+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555741+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673400,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:00.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2024-09-12T16:31:00.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:01.277 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:01.277 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:01.289 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1593,"stamp":"2024-09-12T16:30:59.424174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":48,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.559958+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.559958+0000","last_peered":"2024-09-12T16:30:46.559958+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.559958+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1090,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555741+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555741+0000","last_peered":"2024-09-12T16:30:46.555741+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555741+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673400,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:01.290 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:01.584 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:01.584 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:01.596 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2024-09-12T16:31:01.424651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002635"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673401,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:01.597 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:01.597 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:01.597 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:01.597 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:01.597 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:01.597 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:01.598 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:01.598 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:01.598 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:01.598 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:01.598 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:01.598 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:01.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:01.964 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:01.964 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2182,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:01.978 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:01.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:02.274 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:02.274 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:02.286 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2024-09-12T16:31:01.424651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002635"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673401,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:05.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:05.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2024-09-12T16:31:05.596 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:05.596 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:05.609 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2024-09-12T16:31:05.425556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002541"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673401,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:05.609 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:05.903 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:05.903 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:05.916 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2024-09-12T16:31:05.425556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002541"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673401,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:05.917 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:05.918 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:05.918 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:05.918 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:05.918 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:05.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:06.284 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:06.284 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2187,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:06.297 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:06.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:06.594 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:06.594 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:06.607 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2024-09-12T16:31:05.425556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002541"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327579+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327579+0000","last_peered":"2024-09-12T16:30:48.327579+0000","last_clean":"2024-09-12T16:30:48.327579+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327579+0000","last_undegraded":"2024-09-12T16:30:48.327579+0000","last_fullsized":"2024-09-12T16:30:48.327579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673401,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:09.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:09.918 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:09.919 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:09.931 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-09-12T16:31:09.426471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002692"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673402,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:09.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:10.228 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:10.229 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:10.241 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-09-12T16:31:09.426471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002692"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673402,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:10.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:10.243 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:10.243 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:10.243 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:10.243 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:10.243 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:10.611 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:10.611 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2191,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:10.625 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:10.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:10.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2024-09-12T16:31:10.918 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:10.918 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:10.930 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-09-12T16:31:09.426471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002692"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673402,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:13.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:14.241 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:14.242 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:14.254 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-09-12T16:31:13.427353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673403,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:14.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:14.551 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:14.551 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:14.563 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-09-12T16:31:13.427353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673403,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:14.564 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:14.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:14.565 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:14.566 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:14.566 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:14.935 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:14.935 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2195,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:14.948 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:14.949 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:15.242 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:15.243 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:15.255 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-09-12T16:31:13.427353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673403,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:16.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2024-09-12T16:31:18.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:18.567 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:18.567 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:18.579 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1602,"stamp":"2024-09-12T16:31:17.428164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673404,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:18.580 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:18.874 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:18.874 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:18.886 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1602,"stamp":"2024-09-12T16:31:17.428164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673404,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:18.887 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:18.887 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:18.887 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:18.887 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:18.887 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:18.888 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:18.888 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:18.888 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:18.888 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:18.888 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:18.888 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:18.888 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:18.889 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:19.259 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:19.260 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2200,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:19.272 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:19.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:19.569 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:19.570 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:19.582 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2024-09-12T16:31:19.428755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002796"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673404,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:21.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2024-09-12T16:31:22.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:22.893 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:22.893 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:22.906 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1604,"stamp":"2024-09-12T16:31:21.429143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673405,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:22.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:23.199 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:23.200 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:23.212 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1604,"stamp":"2024-09-12T16:31:21.429143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673405,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:23.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:23.213 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:23.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:23.213 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:23.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:23.213 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:23.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:23.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:23.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:23.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:23.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:23.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:23.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:23.582 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:23.582 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2204,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:23.595 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:23.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:23.888 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:23.901 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2024-09-12T16:31:23.429656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002720"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673405,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:26.904 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:27.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2024-09-12T16:31:27.215 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:27.215 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:27.227 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1606,"stamp":"2024-09-12T16:31:25.430033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002679"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673405,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:27.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:27.522 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:27.535 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2024-09-12T16:31:27.430451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002661"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673406,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:27.536 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:27.536 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:27.536 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:27.536 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:27.536 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:27.536 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:27.537 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:27.537 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:27.537 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:27.537 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:27.537 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:27.537 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:27.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:27.905 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:27.905 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2208,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:27.918 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:27.918 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:28.213 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:28.214 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:28.226 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2024-09-12T16:31:27.430451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002661"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673406,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:31.228 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:31.539 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:31.539 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:31.552 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-09-12T16:31:31.431497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002741"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673407,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:31.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:31.847 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:31.847 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:31.859 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-09-12T16:31:31.431497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002741"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673407,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:31.860 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:31.860 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:31.860 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:31.860 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:31.860 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:31.860 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:31.861 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:31.861 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:31.861 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:31.861 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:31.861 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:31.861 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:31.861 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:32.228 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:32.228 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2212,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:32.241 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:32.241 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:32.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2024-09-12T16:31:32.536 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:32.537 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:32.549 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-09-12T16:31:31.431497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002741"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673407,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:35.551 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:35.858 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:35.858 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:35.871 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2024-09-12T16:31:35.432476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002820"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673407,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:35.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:36.166 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:36.167 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:36.179 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2024-09-12T16:31:35.432476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002820"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673407,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:36.180 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:36.181 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:36.182 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:36.182 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:36.182 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:36.551 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:36.551 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2217,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:36.564 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:36.564 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:36.858 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:36.859 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:36.871 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2024-09-12T16:31:35.432476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002820"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673407,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:37.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2024-09-12T16:31:39.873 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:40.182 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:40.183 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:40.195 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-09-12T16:31:39.433430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673408,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:40.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-09-12T16:31:40.498+0000 7fbb71bb2640 0 --2- 172.21.15.19:0/865766654 >> v2:172.21.15.19:3300/0 conn(0x7fbb6c0aada0 0x7fbb6c0ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-12T16:31:40.689 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:40.690 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:40.702 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-09-12T16:31:39.433430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673408,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:40.703 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:40.703 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:40.703 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:40.703 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:40.703 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:40.703 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:40.704 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:40.704 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:40.704 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:40.704 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:40.704 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:40.704 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:40.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:41.068 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:41.069 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2221,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:41.081 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:41.081 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:41.377 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:41.377 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:41.389 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-09-12T16:31:39.433430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673408,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:43.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2024-09-12T16:31:44.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:44.703 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:44.703 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:44.715 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1615,"stamp":"2024-09-12T16:31:43.434326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002828"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673409,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:44.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:45.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:45.013 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:45.025 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1615,"stamp":"2024-09-12T16:31:43.434326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002828"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673409,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:45.026 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:45.026 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:45.026 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:45.026 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:45.027 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:45.027 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:45.393 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:45.394 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2226,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:45.407 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:45.407 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:45.704 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:45.705 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:45.717 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1616,"stamp":"2024-09-12T16:31:45.434869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002830"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673410,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:48.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2024-09-12T16:31:48.718 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:49.026 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:49.027 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:49.039 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1617,"stamp":"2024-09-12T16:31:47.435340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673410,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:49.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:49.336 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:49.337 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:49.349 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1617,"stamp":"2024-09-12T16:31:47.435340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673410,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:49.350 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:49.350 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563649+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563649+0000', 'last_peered': '2024-09-12T16:30:46.563649+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563649+0000', 'last_undegraded': '2024-09-12T16:30:46.563649+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:49.350 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:49.350 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1021, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560979+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.560979+0000', 'last_peered': '2024-09-12T16:30:46.560979+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.560979+0000', 'last_undegraded': '2024-09-12T16:30:46.560979+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:49.351 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:49.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:49.719 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:49.719 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2230,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:49.732 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:49.732 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:50.029 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:50.030 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:50.042 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1618,"stamp":"2024-09-12T16:31:49.435834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002977"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563649+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563649+0000","last_peered":"2024-09-12T16:30:46.563649+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563649+0000","last_undegraded":"2024-09-12T16:30:46.563649+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560979+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.560979+0000","last_peered":"2024-09-12T16:30:46.560979+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.560979+0000","last_undegraded":"2024-09-12T16:30:46.560979+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673410,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:53.044 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:53.354 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:53.354 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:53.367 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1619,"stamp":"2024-09-12T16:31:51.436414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002983"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673411,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:53.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:53.663 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:53.663 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:53.676 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2024-09-12T16:31:53.436987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673411,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:53.676 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:53.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:53.678 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:53.678 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:53.678 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:54.046 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:54.046 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2234,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:54.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2024-09-12T16:31:54.059 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:54.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:54.354 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:54.355 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:54.367 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2024-09-12T16:31:53.436987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673411,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:57.369 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:57.679 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:57.679 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:57.692 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-09-12T16:31:57.438071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673412,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:57.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:57.988 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:57.988 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:58.000 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-09-12T16:31:57.438071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673412,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:58.001 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:31:58.001 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:31:58.002 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:31:58.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:31:58.372 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:58.372 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2239,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:31:58.384 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:31:58.384 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:31:58.682 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:31:58.683 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:31:58.694 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-09-12T16:31:57.438071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673412,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:31:59.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2024-09-12T16:32:01.696 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:02.006 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:02.006 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:02.019 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-09-12T16:32:01.438960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673413,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:02.020 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:02.312 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:02.312 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:02.324 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-09-12T16:32:01.438960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673413,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:02.325 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:02.325 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:02.325 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:02.325 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:02.325 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:02.325 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:02.326 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:02.326 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:02.326 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:02.326 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:02.326 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:02.326 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:02.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:02.694 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:02.694 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2243,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:02.707 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:02.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:03.001 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:03.001 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:03.013 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-09-12T16:32:01.438960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673413,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:04.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2024-09-12T16:32:06.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:06.325 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:06.325 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:06.337 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-09-12T16:32:05.440007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673414,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:06.338 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:06.633 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:06.634 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:06.646 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-09-12T16:32:05.440007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673414,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:06.647 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:06.648 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:06.648 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:06.648 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:07.012 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:07.013 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2247,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:07.025 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:07.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:07.321 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:07.321 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:07.334 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-09-12T16:32:05.440007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673414,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:10.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2024-09-12T16:32:10.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:10.644 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:10.644 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:10.656 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2024-09-12T16:32:09.441030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002958"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673414,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:10.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:10.949 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:10.950 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:10.962 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2024-09-12T16:32:09.441030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002958"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673414,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:10.963 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:10.964 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:10.964 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:10.964 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:10.964 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:10.964 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:11.336 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:11.336 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2252,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:11.349 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:11.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:11.643 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:11.644 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:11.655 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2024-09-12T16:32:11.441511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673415,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:14.657 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:14.965 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:14.966 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:14.979 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2024-09-12T16:32:13.442078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003118"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673415,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:14.980 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:15.275 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:15.276 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:15.287 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2024-09-12T16:32:13.442078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003118"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673415,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:15.288 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:15.288 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:15.288 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:15.289 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:15.290 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:15.290 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:15.656 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:15.656 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2256,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:15.669 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:15.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:15.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2024-09-12T16:32:15.963 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:15.963 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:15.976 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1631,"stamp":"2024-09-12T16:32:15.442547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003076"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673416,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:18.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:19.288 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:19.288 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:19.300 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1632,"stamp":"2024-09-12T16:32:17.442954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002947"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673416,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:19.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:19.596 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:19.596 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:19.609 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2024-09-12T16:32:19.443417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002802"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673416,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:19.609 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:19.609 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:19.610 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:19.611 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:19.611 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:19.978 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:19.978 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2260,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:19.991 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:19.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:20.285 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:20.285 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:20.297 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2024-09-12T16:32:19.443417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002802"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673416,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:21.078 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2024-09-12T16:32:23.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:23.610 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:23.611 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:23.623 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-09-12T16:32:23.444576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003064"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673417,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:23.624 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:23.918 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:23.930 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-09-12T16:32:23.444576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003064"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673417,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:23.931 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:23.932 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:23.933 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:23.933 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:23.933 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:23.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:24.301 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:24.301 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2265,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:24.314 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:24.314 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:24.608 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:24.609 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:24.621 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-09-12T16:32:23.444576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003064"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327595+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327595+0000","last_peered":"2024-09-12T16:30:48.327595+0000","last_clean":"2024-09-12T16:30:48.327595+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327595+0000","last_undegraded":"2024-09-12T16:30:48.327595+0000","last_fullsized":"2024-09-12T16:30:48.327595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673417,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:26.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2024-09-12T16:32:27.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:27.931 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:27.944 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-09-12T16:32:27.445582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003035"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673418,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:27.945 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:28.238 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:28.238 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:28.251 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-09-12T16:32:27.445582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003035"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673418,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:28.251 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.559975+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.559975+0000', 'last_peered': '2024-09-12T16:30:46.559975+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.559975+0000', 'last_undegraded': '2024-09-12T16:30:46.559975+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:28.252 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:28.253 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:28.253 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:28.253 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:28.618 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:28.618 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2269,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:28.631 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:28.631 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:28.925 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:28.925 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:28.937 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-09-12T16:32:27.445582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003035"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.559975+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.559975+0000","last_peered":"2024-09-12T16:30:46.559975+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.559975+0000","last_undegraded":"2024-09-12T16:30:46.559975+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559391+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559391+0000","last_peered":"2024-09-12T16:30:46.559391+0000","last_clean":"2024-09-12T16:30:46.559391+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559391+0000","last_undegraded":"2024-09-12T16:30:46.559391+0000","last_fullsized":"2024-09-12T16:30:46.559391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673418,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:31.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2024-09-12T16:32:31.939 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:32.252 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:32.253 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:32.265 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-09-12T16:32:31.446445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673419,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:32.266 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:32.563 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:32.564 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:32.577 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-09-12T16:32:31.446445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673419,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561292+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561292+0000', 'last_peered': '2024-09-12T16:30:46.561292+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561292+0000', 'last_undegraded': '2024-09-12T16:30:46.561292+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:32.578 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:32.579 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:32.579 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:32.579 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:32.579 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:32.579 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:32.948 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:32.948 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2273,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:32.961 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:32.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:33.257 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:33.257 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:33.269 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-09-12T16:32:31.446445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561292+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561292+0000","last_peered":"2024-09-12T16:30:46.561292+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561292+0000","last_undegraded":"2024-09-12T16:30:46.561292+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673419,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:36.271 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:36.580 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:36.580 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:36.592 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-09-12T16:32:35.447298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673420,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:36.593 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:36.887 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:36.887 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:36.900 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-09-12T16:32:35.447298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673420,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:36.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:36.902 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:36.902 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:36.902 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:36.902 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:36.902 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:37.275 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:37.275 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2278,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:37.288 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:37.288 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:37.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2024-09-12T16:32:37.585 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:37.585 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:37.597 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1642,"stamp":"2024-09-12T16:32:37.447675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002533"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673420,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:40.599 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:40.912 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:40.912 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:40.924 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-09-12T16:32:39.448069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002486"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673420,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:40.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:41.218 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:41.219 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:41.231 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-09-12T16:32:39.448069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002486"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673420,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:41.232 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:41.233 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:41.233 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:41.233 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:41.233 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:41.600 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:41.600 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2282,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:41.613 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:41.613 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:41.909 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:41.910 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:41.920 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1644,"stamp":"2024-09-12T16:32:41.448519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002558"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673421,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:42.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2024-09-12T16:32:44.921 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:45.246 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:45.247 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:45.256 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2024-09-12T16:32:43.448999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673421,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:45.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:45.554 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:45.554 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:45.567 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2024-09-12T16:32:45.449437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002490"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673422,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:45.568 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:45.568 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:45.568 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:45.568 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:45.568 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:45.568 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:45.568 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:45.569 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:45.569 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:45.569 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:45.569 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:45.569 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:45.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:45.938 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:45.939 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2286,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:45.952 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:45.952 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:46.250 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:46.250 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:46.262 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2024-09-12T16:32:45.449437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002490"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673422,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:48.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2024-09-12T16:32:49.264 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:49.572 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:49.572 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:49.585 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2024-09-12T16:32:49.450440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002764"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673422,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:49.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:49.878 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:49.879 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:49.891 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2024-09-12T16:32:49.450440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002764"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673422,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:49.891 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:49.892 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:49.893 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:49.893 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:49.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:50.261 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:50.311 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2291,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:50.312 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:50.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:50.607 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:50.607 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:50.620 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2024-09-12T16:32:49.450440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002764"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673422,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:53.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2024-09-12T16:32:53.622 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:53.931 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:53.931 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:53.943 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-09-12T16:32:53.451252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673423,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:53.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:54.241 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:54.241 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:54.254 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-09-12T16:32:53.451252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673423,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:54.254 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:54.255 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:54.256 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:54.256 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:54.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:54.631 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:54.631 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2295,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:54.645 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:54.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:54.940 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:54.940 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:54.952 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-09-12T16:32:53.451252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673423,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:57.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:58.265 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:58.265 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:58.278 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-09-12T16:32:57.452438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673424,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:58.279 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:58.571 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:58.571 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:58.584 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-09-12T16:32:57.452438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673424,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:32:58.584 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:32:58.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:58.586 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:32:58.586 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1091, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.555837+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:30:46.555837+0000', 'last_peered': '2024-09-12T16:30:46.555837+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:30:46.555837+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:32:58.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:32:58.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2024-09-12T16:32:58.958 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:58.959 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2299,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:32:58.972 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:32:58.972 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:32:59.268 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:32:59.268 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:32:59.281 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-09-12T16:32:57.452438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1091,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.555837+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:30:46.555837+0000","last_peered":"2024-09-12T16:30:46.555837+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:30:46.555837+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673424,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:02.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:02.590 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:02.591 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:02.604 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2024-09-12T16:33:01.453287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002847"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673425,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:02.604 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:02.899 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:02.900 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:02.912 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2024-09-12T16:33:01.453287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002847"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673425,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:02.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:02.914 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:02.914 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 49, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:30:46.560090+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:30:46.560090+0000', 'last_peered': '2024-09-12T16:30:46.560090+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:30:46.560090+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:02.914 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:02.914 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:02.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:03.280 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:03.353 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2304,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:03.358 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:03.358 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:03.655 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:03.655 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:03.668 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1655,"stamp":"2024-09-12T16:33:03.453695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002851"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":49,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:30:46.560090+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:30:46.560090+0000","last_peered":"2024-09-12T16:30:46.560090+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:30:46.560090+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673425,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:04.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2024-09-12T16:33:06.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:06.984 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:06.985 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:06.994 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2024-09-12T16:33:05.454127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002875"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673426,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:06.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:07.291 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:07.292 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:07.304 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2024-09-12T16:33:05.454127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002875"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327634+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:30:48.327634+0000","last_peered":"2024-09-12T16:30:48.327634+0000","last_clean":"2024-09-12T16:30:48.327634+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:30:48.327634+0000","last_undegraded":"2024-09-12T16:30:48.327634+0000","last_fullsized":"2024-09-12T16:30:48.327634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673426,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:07.305 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:07.305 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:07.305 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:07.305 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:07.305 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:07.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:07.306 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:07.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:07.306 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:07.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:07.306 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:07.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:07.306 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:07.676 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:07.676 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2308,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:07.690 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:07.690 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:07.988 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:07.988 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:08.000 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2024-09-12T16:33:07.454622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002833"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673426,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:09.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2024-09-12T16:33:11.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:11.310 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:11.310 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:11.323 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-09-12T16:33:09.455135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002697"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673426,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:11.324 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:11.619 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:11.619 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:11.631 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2024-09-12T16:33:11.455543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673427,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:11.632 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:11.632 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:11.632 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:11.633 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:11.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:12.004 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:12.004 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2312,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:12.017 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:12.017 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:12.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:12.315 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:12.327 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2024-09-12T16:33:11.455543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673427,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:15.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2024-09-12T16:33:15.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:15.639 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:15.639 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:15.653 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2024-09-12T16:33:15.456291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002595"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673428,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:15.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:15.946 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:15.946 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:15.959 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2024-09-12T16:33:15.456291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002595"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673428,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:15.959 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:15.960 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:15.961 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:15.961 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:16.335 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:16.335 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2317,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:16.348 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:16.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:16.643 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:16.644 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:16.656 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2024-09-12T16:33:15.456291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002595"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673428,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:19.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:19.970 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:19.970 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:19.982 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-09-12T16:33:19.457256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673428,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:19.983 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:20.280 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:20.384 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:20.384 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-09-12T16:33:19.457256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673428,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:20.385 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:20.385 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:20.385 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:20.385 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:20.385 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:20.386 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:20.386 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:20.386 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:20.386 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:20.386 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:20.386 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:20.386 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:20.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:20.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2024-09-12T16:33:20.760 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:20.760 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2321,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:20.773 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:20.773 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:21.066 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:21.066 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:21.078 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-09-12T16:33:19.457256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673428,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:24.080 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:24.390 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:24.390 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:24.402 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-09-12T16:33:23.458236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673429,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:24.403 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:24.698 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:24.699 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:24.711 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-09-12T16:33:23.458236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673429,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:24.712 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:24.713 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:24.713 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:24.713 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:24.713 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:24.713 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:25.078 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:25.079 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2325,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:25.092 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:25.092 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:25.386 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:25.398 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-09-12T16:33:23.458236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673429,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:25.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2024-09-12T16:33:28.401 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:28.713 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:28.713 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:28.726 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-09-12T16:33:27.459034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002743"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673430,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:28.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:29.021 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:29.021 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:29.033 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-09-12T16:33:27.459034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002743"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673430,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:29.034 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:29.035 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:29.036 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:29.036 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:29.036 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:29.404 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:29.404 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2330,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:29.417 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:29.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:29.717 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:29.718 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:29.729 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1668,"stamp":"2024-09-12T16:33:29.459473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002764"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673430,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:31.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2024-09-12T16:33:32.731 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:33.039 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:33.039 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:33.052 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-09-12T16:33:31.460045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002788"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673431,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:33.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:33.346 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:33.347 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:33.359 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-09-12T16:33:31.460045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002788"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673431,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:33.359 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:33.359 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:33.360 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:33.361 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:33.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:33.728 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:33.728 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2334,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:33.741 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:33.741 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:34.034 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:34.034 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:34.046 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1670,"stamp":"2024-09-12T16:33:33.460531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002794"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673431,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:36.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2024-09-12T16:33:37.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:37.362 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:37.363 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:37.376 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2024-09-12T16:33:35.461004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673432,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:37.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:37.676 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:37.894 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:37.894 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2024-09-12T16:33:37.461429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002778"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673432,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:37.895 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:37.895 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:37.896 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:37.896 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:37.896 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:37.896 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:37.897 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:37.897 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:37.897 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:37.897 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:37.897 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:37.897 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:37.898 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:38.276 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:38.276 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2339,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:38.289 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:38.290 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:38.584 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:38.584 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:38.596 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2024-09-12T16:33:37.461429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002778"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673432,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:41.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:41.908 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:41.909 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:41.921 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2024-09-12T16:33:41.462425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:41.922 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:42.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2024-09-12T16:33:42.216 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:42.216 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:42.229 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2024-09-12T16:33:41.462425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:42.229 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:42.230 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:42.231 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:42.231 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:42.231 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:42.601 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:42.602 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2343,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:42.615 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:42.616 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:42.911 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:42.911 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:42.924 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2024-09-12T16:33:41.462425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:45.926 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:46.234 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:46.235 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:46.246 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-09-12T16:33:45.463332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673434,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:46.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:46.544 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:46.544 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:46.557 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-09-12T16:33:45.463332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673434,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:46.558 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:46.558 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:46.558 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:46.558 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:46.558 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:46.558 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:46.558 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:46.559 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:46.559 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:46.559 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:46.559 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:46.559 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:46.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:46.930 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:46.931 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2347,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:46.944 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:46.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:47.241 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:47.241 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:47.253 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-09-12T16:33:45.463332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673434,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:47.609 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2024-09-12T16:33:50.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:50.567 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:50.568 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:50.580 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-09-12T16:33:49.464203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002773"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673434,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:50.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:50.876 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:50.876 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:50.889 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-09-12T16:33:49.464203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002773"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673434,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:50.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:50.890 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:50.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:50.890 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:50.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:50.890 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:50.890 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:50.891 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:50.891 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:50.891 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:50.891 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:50.891 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:50.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:51.259 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:51.259 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2352,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:51.273 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:51.273 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:51.580 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1679,"stamp":"2024-09-12T16:33:51.464789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002804"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673435,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:53.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2024-09-12T16:33:54.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:54.893 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:54.893 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:54.906 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-09-12T16:33:53.465221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673435,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:54.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:55.200 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:55.200 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:55.212 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-09-12T16:33:53.465221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673435,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:55.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2024-09-12T16:33:55.213 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.563974+0000', 'last_change': '2024-09-12T16:28:54.589031+0000', 'last_active': '2024-09-12T16:30:46.563974+0000', 'last_peered': '2024-09-12T16:30:46.563974+0000', 'last_clean': '2024-09-12T16:28:49.457233+0000', 'last_became_active': '2024-09-12T16:28:54.589031+0000', 'last_became_peered': '2024-09-12T16:28:54.589031+0000', 'last_unstale': '2024-09-12T16:30:46.563974+0000', 'last_undegraded': '2024-09-12T16:30:46.563974+0000', 'last_fullsized': '2024-09-12T16:28:54.560881+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000873738, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:55.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1022, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561073+0000', 'last_change': '2024-09-12T16:28:54.570323+0000', 'last_active': '2024-09-12T16:30:46.561073+0000', 'last_peered': '2024-09-12T16:30:46.561073+0000', 'last_clean': '2024-09-12T16:28:49.457721+0000', 'last_became_active': '2024-09-12T16:28:54.570323+0000', 'last_became_peered': '2024-09-12T16:28:54.570323+0000', 'last_unstale': '2024-09-12T16:30:46.561073+0000', 'last_undegraded': '2024-09-12T16:30:46.561073+0000', 'last_fullsized': '2024-09-12T16:28:54.557486+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000938769, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.561378+0000', 'last_change': '2024-09-12T16:28:54.578074+0000', 'last_active': '2024-09-12T16:30:46.561378+0000', 'last_peered': '2024-09-12T16:30:46.561378+0000', 'last_clean': '2024-09-12T16:28:49.457676+0000', 'last_became_active': '2024-09-12T16:28:54.578074+0000', 'last_became_peered': '2024-09-12T16:28:54.578074+0000', 'last_unstale': '2024-09-12T16:30:46.561378+0000', 'last_undegraded': '2024-09-12T16:30:46.561378+0000', 'last_fullsized': '2024-09-12T16:28:54.560589+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001087616, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1023, 'reported_epoch': 509, 'state': 'active+undersized', 'last_fresh': '2024-09-12T16:30:46.560079+0000', 'last_change': '2024-09-12T16:28:54.560998+0000', 'last_active': '2024-09-12T16:30:46.560079+0000', 'last_peered': '2024-09-12T16:30:46.560079+0000', 'last_clean': '2024-09-12T16:28:49.456817+0000', 'last_became_active': '2024-09-12T16:28:54.560998+0000', 'last_became_peered': '2024-09-12T16:28:54.560998+0000', 'last_unstale': '2024-09-12T16:30:46.560079+0000', 'last_undegraded': '2024-09-12T16:30:46.560079+0000', 'last_fullsized': '2024-09-12T16:28:54.553266+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000614855, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "18'2", 'reported_seq': 50, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429370+0000', 'last_change': '2024-09-12T16:28:54.573412+0000', 'last_active': '2024-09-12T16:31:00.429370+0000', 'last_peered': '2024-09-12T16:31:00.429370+0000', 'last_clean': '2024-09-12T15:54:57.991815+0000', 'last_became_active': '2024-09-12T16:28:54.573412+0000', 'last_became_peered': '2024-09-12T16:28:54.573412+0000', 'last_unstale': '2024-09-12T16:31:00.429370+0000', 'last_undegraded': '2024-09-12T16:28:54.559031+0000', 'last_fullsized': '2024-09-12T16:28:54.558958+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:54.979009+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000519456, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 1}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2024-09-12T16:33:55.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "13'52", 'reported_seq': 1092, 'reported_epoch': 509, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-12T16:31:00.429322+0000', 'last_change': '2024-09-12T16:28:54.556824+0000', 'last_active': '2024-09-12T16:31:00.429322+0000', 'last_peered': '2024-09-12T16:31:00.429322+0000', 'last_clean': '2024-09-12T16:28:49.455934+0000', 'last_became_active': '2024-09-12T16:28:54.556824+0000', 'last_became_peered': '2024-09-12T16:28:54.556824+0000', 'last_unstale': '2024-09-12T16:31:00.429322+0000', 'last_undegraded': '2024-09-12T16:28:54.552590+0000', 'last_fullsized': '2024-09-12T16:28:54.552507+0000', 'mapping_epoch': 498, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'last_clean_scrub_stamp': '2024-09-12T15:54:51.964822+0000', 'objects_scrubbed': 0, 'log_size': 52, 'log_dups_size': 0, 'ondisk_log_size': 52, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 56, 'num_read_kb': 46, 'num_write': 79, 'num_write_kb': 1106, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-12T16:33:55.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2024-09-12T16:33:55.582 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:55.582 INFO:teuthology.orchestra.run.smithi019.stdout:{"fsid":"c66699a9-7576-4b7c-b56e-924785d0fe22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/8 objects degraded (37.500%), 2 pgs degraded, 6 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2356,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":509,"num_osds":3,"num_up_osds":2,"osd_up_since":1726158534,"num_in_osds":3,"osd_in_since":1726156480,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":2}],"num_pgs":9,"num_pools":2,"num_objects":4,"data_bytes":590387,"bytes_used":3514908672,"bytes_avail":286395383808,"bytes_total":289910292480,"degraded_objects":3,"degraded_total":8,"degraded_ratio":0.375},"fsmap":{"epoch":1,"btime":"2024-09-12T15:54:39:237072+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2024-09-12T16:30:49.422047+0000","services":{}},"progress_events":{}} 2024-09-12T16:33:55.594 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2024-09-12T16:33:55.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:33:55.889 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-09-12T16:33:55.890 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-09-12T16:33:55.902 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":1681,"stamp":"2024-09-12T16:33:55.465770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3432528,"kb_used_data":38448,"kb_used_omap":290,"kb_used_meta":3379613,"kb_avail":279682992,"statfs":{"total":289910292480,"available":286395383808,"internally_reserved":0,"allocated":39370752,"data_stored":30130815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":297610,"internal_metadata":3460724086},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.563974+0000","last_change":"2024-09-12T16:28:54.589031+0000","last_active":"2024-09-12T16:30:46.563974+0000","last_peered":"2024-09-12T16:30:46.563974+0000","last_clean":"2024-09-12T16:28:49.457233+0000","last_became_active":"2024-09-12T16:28:54.589031+0000","last_became_peered":"2024-09-12T16:28:54.589031+0000","last_unstale":"2024-09-12T16:30:46.563974+0000","last_undegraded":"2024-09-12T16:30:46.563974+0000","last_fullsized":"2024-09-12T16:28:54.560881+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:07:34.388212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087373799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1022,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561073+0000","last_change":"2024-09-12T16:28:54.570323+0000","last_active":"2024-09-12T16:30:46.561073+0000","last_peered":"2024-09-12T16:30:46.561073+0000","last_clean":"2024-09-12T16:28:49.457721+0000","last_became_active":"2024-09-12T16:28:54.570323+0000","last_became_peered":"2024-09-12T16:28:54.570323+0000","last_unstale":"2024-09-12T16:30:46.561073+0000","last_undegraded":"2024-09-12T16:30:46.561073+0000","last_fullsized":"2024-09-12T16:28:54.557486+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:19:51.249520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000938769,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.561378+0000","last_change":"2024-09-12T16:28:54.578074+0000","last_active":"2024-09-12T16:30:46.561378+0000","last_peered":"2024-09-12T16:30:46.561378+0000","last_clean":"2024-09-12T16:28:49.457676+0000","last_became_active":"2024-09-12T16:28:54.578074+0000","last_became_peered":"2024-09-12T16:28:54.578074+0000","last_unstale":"2024-09-12T16:30:46.561378+0000","last_undegraded":"2024-09-12T16:30:46.561378+0000","last_fullsized":"2024-09-12T16:28:54.560589+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:09:33.808712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001087616,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1023,"reported_epoch":509,"state":"active+undersized","last_fresh":"2024-09-12T16:30:46.560079+0000","last_change":"2024-09-12T16:28:54.560998+0000","last_active":"2024-09-12T16:30:46.560079+0000","last_peered":"2024-09-12T16:30:46.560079+0000","last_clean":"2024-09-12T16:28:49.456817+0000","last_became_active":"2024-09-12T16:28:54.560998+0000","last_became_peered":"2024-09-12T16:28:54.560998+0000","last_unstale":"2024-09-12T16:30:46.560079+0000","last_undegraded":"2024-09-12T16:30:46.560079+0000","last_fullsized":"2024-09-12T16:28:54.553266+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:45:23.836991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061485499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":50,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429370+0000","last_change":"2024-09-12T16:28:54.573412+0000","last_active":"2024-09-12T16:31:00.429370+0000","last_peered":"2024-09-12T16:31:00.429370+0000","last_clean":"2024-09-12T15:54:57.991815+0000","last_became_active":"2024-09-12T16:28:54.573412+0000","last_became_peered":"2024-09-12T16:28:54.573412+0000","last_unstale":"2024-09-12T16:31:00.429370+0000","last_undegraded":"2024-09-12T16:28:54.559031+0000","last_fullsized":"2024-09-12T16:28:54.558958+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:26:09.689437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051945599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:48.327646+0000","last_change":"2024-09-12T15:54:58.074779+0000","last_active":"2024-09-12T16:30:48.327646+0000","last_peered":"2024-09-12T16:30:48.327646+0000","last_clean":"2024-09-12T16:30:48.327646+0000","last_became_active":"2024-09-12T15:54:55.993377+0000","last_became_peered":"2024-09-12T15:54:55.993377+0000","last_unstale":"2024-09-12T16:30:48.327646+0000","last_undegraded":"2024-09-12T16:30:48.327646+0000","last_fullsized":"2024-09-12T16:30:48.327646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:56:53.706377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040547700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:31:07.000044+0000","last_change":"2024-09-12T15:54:58.074740+0000","last_active":"2024-09-12T16:31:07.000044+0000","last_peered":"2024-09-12T16:31:07.000044+0000","last_clean":"2024-09-12T16:31:07.000044+0000","last_became_active":"2024-09-12T15:54:55.992768+0000","last_became_peered":"2024-09-12T15:54:55.992768+0000","last_unstale":"2024-09-12T16:31:07.000044+0000","last_undegraded":"2024-09-12T16:31:07.000044+0000","last_fullsized":"2024-09-12T16:31:07.000044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:27.647726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040474499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-09-12T16:30:46.559441+0000","last_change":"2024-09-12T15:54:57.995557+0000","last_active":"2024-09-12T16:30:46.559441+0000","last_peered":"2024-09-12T16:30:46.559441+0000","last_clean":"2024-09-12T16:30:46.559441+0000","last_became_active":"2024-09-12T15:54:55.992654+0000","last_became_peered":"2024-09-12T15:54:55.992654+0000","last_unstale":"2024-09-12T16:30:46.559441+0000","last_undegraded":"2024-09-12T16:30:46.559441+0000","last_fullsized":"2024-09-12T16:30:46.559441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:54.979009+0000","last_clean_scrub_stamp":"2024-09-12T15:54:54.979009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:00:57.122701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054289799999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":1092,"reported_epoch":509,"state":"active+undersized+degraded","last_fresh":"2024-09-12T16:31:00.429322+0000","last_change":"2024-09-12T16:28:54.556824+0000","last_active":"2024-09-12T16:31:00.429322+0000","last_peered":"2024-09-12T16:31:00.429322+0000","last_clean":"2024-09-12T16:28:49.455934+0000","last_became_active":"2024-09-12T16:28:54.556824+0000","last_became_peered":"2024-09-12T16:28:54.556824+0000","last_unstale":"2024-09-12T16:31:00.429322+0000","last_undegraded":"2024-09-12T16:28:54.552590+0000","last_fullsized":"2024-09-12T16:28:54.552507+0000","mapping_epoch":498,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:51.964822+0000","last_clean_scrub_stamp":"2024-09-12T15:54:51.964822+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T03:50:24.300130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673436,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119012,"kb_used_data":6480,"kb_used_omap":83,"kb_used_meta":1112428,"kb_avail":93252828,"statfs":{"total":96636764160,"available":95490895872,"internally_reserved":0,"allocated":6635520,"data_stored":3597210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":85492,"internal_metadata":1139126796},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673373,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1379856,"kb_used_data":26072,"kb_used_omap":104,"kb_used_meta":1339543,"kb_avail":92991984,"statfs":{"total":96636764160,"available":95223791616,"internally_reserved":0,"allocated":26697728,"data_stored":23526782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107352,"internal_metadata":1371692200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933660,"kb_used_data":5896,"kb_used_omap":102,"kb_used_meta":927641,"kb_avail":93438180,"statfs":{"total":96636764160,"available":95680696320,"internally_reserved":0,"allocated":6037504,"data_stored":3006823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104766,"internal_metadata":949905090},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:33:58.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2024-09-12T16:33:58.419 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-09-12T16:33:58.822 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2024-09-12T16:33:58.823 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2024-09-12T16:33:58.823 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2024-09-12T16:33:58.823 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2024-09-12T16:33:58.824 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:33:58.823+0000 7ff7061bd640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 32713) UID: 0 2024-09-12T16:33:58.824 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:33:58.823+0000 7ff7061bd640 -1 osd.1 509 *** Got signal Terminated *** 2024-09-12T16:33:58.824 INFO:tasks.ceph.cluster2.osd.1.smithi019.stderr:2024-09-12T16:33:58.823+0000 7ff7061bd640 -1 osd.1 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T16:33:58.825 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:33:58.823+0000 7f766cfd3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 32716) UID: 0 2024-09-12T16:33:58.825 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:33:58.823+0000 7f766cfd3640 -1 osd.2 509 *** Got signal Terminated *** 2024-09-12T16:33:58.825 INFO:tasks.ceph.cluster2.osd.2.smithi019.stderr:2024-09-12T16:33:58.823+0000 7f766cfd3640 -1 osd.2 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-12T16:33:58.825 INFO:tasks.ceph.cluster2.mon.a.smithi019.stderr:2024-09-12T16:33:58.823+0000 7f12569bf640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 32405) UID: 0 2024-09-12T16:33:58.825 INFO:tasks.ceph.cluster2.mon.a.smithi019.stderr:2024-09-12T16:33:58.823+0000 7f12569bf640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-12T16:33:58.826 INFO:tasks.ceph.cluster2.mgr.x.smithi016.stderr:2024-09-12T16:33:58.825+0000 7f90cf7c5640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 36785) UID: 0 2024-09-12T16:33:58.826 INFO:tasks.ceph.cluster2.mgr.x.smithi016.stderr:2024-09-12T16:33:58.825+0000 7f90cf7c5640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-12T16:33:58.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-12T16:35:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-09-12T16:35:58.943 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph.py", line 1990, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2867, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2803, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2464, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1696, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1687, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-09-12T16:35:58.946 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-12T16:35:58.946 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-12T16:35:58.947 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-12T16:35:58.947 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:35:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-09-12T16:35:58.947 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph.py", line 1506, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph.py", line 1990, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2867, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2803, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2464, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1696, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1687, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2024-09-12T16:35:58.948 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-12T16:35:58.948 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-12T16:35:58.948 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:35:58.948 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-12T16:35:58.949 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-12T16:35:58.949 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:35:58.949 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-12T16:35:58.949 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-12T16:35:58.949 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-12T16:35:58.949 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:35:58.949 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-12T16:35:58.950 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-12T16:35:58.950 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-12T16:35:58.950 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:35:58.950 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-12T16:35:58.950 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-12T16:35:58.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T16:35:58.986 INFO:teuthology.orchestra.run.smithi019.stdout:2024-09-12T16:28:54.486300+0000 mon.a (mon.0) 1595 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-12T16:35:58.987 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-12T16:35:58.987 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T16:35:59.059 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T16:35:59.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[WRN\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T16:35:59.205 INFO:teuthology.orchestra.run.smithi019.stdout:2024-09-12T16:28:54.486300+0000 mon.a (mon.0) 1595 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-12T16:35:59.205 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T16:35:59.206 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-12T16:35:59.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T16:35:59.436 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-12T16:35:59.589 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T16:35:59.589 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-12T16:35:59.760 INFO:tasks.ceph:Archiving mon data... 2024-09-12T16:35:59.761 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/data/cluster2.mon.a.tgz 2024-09-12T16:35:59.762 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-09-12T16:35:59.777 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.0FrMTeovij 2024-09-12T16:35:59.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.0FrMTeovij 2024-09-12T16:35:59.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.0FrMTeovij 2024-09-12T16:36:00.030 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.0FrMTeovij is 440KB 2024-09-12T16:36:00.128 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.0FrMTeovij 2024-09-12T16:36:00.144 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-12T16:36:00.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T16:36:00.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T16:36:00.210 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph.py", line 1990, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2867, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2803, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2464, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1696, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1687, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-09-12T16:36:00.211 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-12T16:36:00.229 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-12T16:36:00.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T16:36:00.540 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T16:36:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T16:36:00.551 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":1713,"stamp":"2024-09-12T16:35:58.769959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456888,"kb_used_data":12280,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658632,"statfs":{"total":289910292480,"available":289442439168,"internally_reserved":0,"allocated":12574720,"data_stored":7269448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":455208271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002834"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501538+0000","last_change":"2024-09-12T15:54:18.505765+0000","last_active":"2024-09-12T16:33:20.501538+0000","last_peered":"2024-09-12T16:33:20.501538+0000","last_clean":"2024-09-12T16:33:20.501538+0000","last_became_active":"2024-09-12T15:54:16.504253+0000","last_became_peered":"2024-09-12T15:54:16.504253+0000","last_unstale":"2024-09-12T16:33:20.501538+0000","last_undegraded":"2024-09-12T16:33:20.501538+0000","last_fullsized":"2024-09-12T16:33:20.501538+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:38:53.254216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001172934,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501518+0000","last_change":"2024-09-12T15:54:18.505043+0000","last_active":"2024-09-12T16:33:20.501518+0000","last_peered":"2024-09-12T16:33:20.501518+0000","last_clean":"2024-09-12T16:33:20.501518+0000","last_became_active":"2024-09-12T15:54:16.504145+0000","last_became_peered":"2024-09-12T15:54:16.504145+0000","last_unstale":"2024-09-12T16:33:20.501518+0000","last_undegraded":"2024-09-12T16:33:20.501518+0000","last_fullsized":"2024-09-12T16:33:20.501518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:15:43.919423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073703799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.491513+0000","last_change":"2024-09-12T15:54:18.505427+0000","last_active":"2024-09-12T16:31:50.491513+0000","last_peered":"2024-09-12T16:31:50.491513+0000","last_clean":"2024-09-12T16:31:50.491513+0000","last_became_active":"2024-09-12T15:54:16.503301+0000","last_became_peered":"2024-09-12T15:54:16.503301+0000","last_unstale":"2024-09-12T16:31:50.491513+0000","last_undegraded":"2024-09-12T16:31:50.491513+0000","last_fullsized":"2024-09-12T16:31:50.491513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:26:05.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079584,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.491506+0000","last_change":"2024-09-12T15:54:18.505017+0000","last_active":"2024-09-12T16:31:50.491506+0000","last_peered":"2024-09-12T16:31:50.491506+0000","last_clean":"2024-09-12T16:31:50.491506+0000","last_became_active":"2024-09-12T15:54:16.504202+0000","last_became_peered":"2024-09-12T15:54:16.504202+0000","last_unstale":"2024-09-12T16:31:50.491506+0000","last_undegraded":"2024-09-12T16:31:50.491506+0000","last_fullsized":"2024-09-12T16:31:50.491506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T02:35:33.901895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007492,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":933,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.349732+0000","last_change":"2024-09-12T15:54:18.508120+0000","last_active":"2024-09-12T16:31:50.349732+0000","last_peered":"2024-09-12T16:31:50.349732+0000","last_clean":"2024-09-12T16:31:50.349732+0000","last_became_active":"2024-09-12T15:54:16.503357+0000","last_became_peered":"2024-09-12T15:54:16.503357+0000","last_unstale":"2024-09-12T16:31:50.349732+0000","last_undegraded":"2024-09-12T16:31:50.349732+0000","last_fullsized":"2024-09-12T16:31:50.349732+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:03:29.295197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001241416,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.458319+0000","last_change":"2024-09-12T15:54:18.830314+0000","last_active":"2024-09-12T16:31:50.458319+0000","last_peered":"2024-09-12T16:31:50.458319+0000","last_clean":"2024-09-12T16:31:50.458319+0000","last_became_active":"2024-09-12T15:54:16.505951+0000","last_became_peered":"2024-09-12T15:54:16.505951+0000","last_unstale":"2024-09-12T16:31:50.458319+0000","last_undegraded":"2024-09-12T16:31:50.458319+0000","last_fullsized":"2024-09-12T16:31:50.458319+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:25:36.672104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039973199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:20.461435+0000","last_change":"2024-09-12T15:54:18.830367+0000","last_active":"2024-09-12T16:32:20.461435+0000","last_peered":"2024-09-12T16:32:20.461435+0000","last_clean":"2024-09-12T16:32:20.461435+0000","last_became_active":"2024-09-12T15:54:16.506142+0000","last_became_peered":"2024-09-12T15:54:16.506142+0000","last_unstale":"2024-09-12T16:32:20.461435+0000","last_undegraded":"2024-09-12T16:32:20.461435+0000","last_fullsized":"2024-09-12T16:32:20.461435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:22:17.633227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043801199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.491489+0000","last_change":"2024-09-12T15:54:18.505434+0000","last_active":"2024-09-12T16:31:50.491489+0000","last_peered":"2024-09-12T16:31:50.491489+0000","last_clean":"2024-09-12T16:31:50.491489+0000","last_became_active":"2024-09-12T15:54:16.505397+0000","last_became_peered":"2024-09-12T15:54:16.505397+0000","last_unstale":"2024-09-12T16:31:50.491489+0000","last_undegraded":"2024-09-12T16:31:50.491489+0000","last_fullsized":"2024-09-12T16:31:50.491489+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:48:51.220899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099554,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":998,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:15.494402+0000","last_change":"2024-09-12T15:54:11.528229+0000","last_active":"2024-09-12T16:32:15.494402+0000","last_peered":"2024-09-12T16:32:15.494402+0000","last_clean":"2024-09-12T16:32:15.494402+0000","last_became_active":"2024-09-12T15:54:11.527071+0000","last_became_peered":"2024-09-12T15:54:11.527071+0000","last_unstale":"2024-09-12T16:32:15.494402+0000","last_undegraded":"2024-09-12T16:32:15.494402+0000","last_fullsized":"2024-09-12T16:32:15.494402+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_clean_scrub_stamp":"2024-09-12T15:54:10.483728+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:21:13.353299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151436,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220404,"statfs":{"total":96636764160,"available":96481693696,"internally_reserved":0,"allocated":3792896,"data_stored":2029558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125656,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246184,"statfs":{"total":96636764160,"available":96508092416,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738873,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179796,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192044,"statfs":{"total":96636764160,"available":96452653056,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:36:00.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T16:36:00.857 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T16:36:00.859 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T16:36:00.871 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":1714,"stamp":"2024-09-12T16:36:00.770480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456888,"kb_used_data":12280,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658632,"statfs":{"total":289910292480,"available":289442439168,"internally_reserved":0,"allocated":12574720,"data_stored":7269448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":455208271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501538+0000","last_change":"2024-09-12T15:54:18.505765+0000","last_active":"2024-09-12T16:33:20.501538+0000","last_peered":"2024-09-12T16:33:20.501538+0000","last_clean":"2024-09-12T16:33:20.501538+0000","last_became_active":"2024-09-12T15:54:16.504253+0000","last_became_peered":"2024-09-12T15:54:16.504253+0000","last_unstale":"2024-09-12T16:33:20.501538+0000","last_undegraded":"2024-09-12T16:33:20.501538+0000","last_fullsized":"2024-09-12T16:33:20.501538+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:38:53.254216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001172934,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501518+0000","last_change":"2024-09-12T15:54:18.505043+0000","last_active":"2024-09-12T16:33:20.501518+0000","last_peered":"2024-09-12T16:33:20.501518+0000","last_clean":"2024-09-12T16:33:20.501518+0000","last_became_active":"2024-09-12T15:54:16.504145+0000","last_became_peered":"2024-09-12T15:54:16.504145+0000","last_unstale":"2024-09-12T16:33:20.501518+0000","last_undegraded":"2024-09-12T16:33:20.501518+0000","last_fullsized":"2024-09-12T16:33:20.501518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:15:43.919423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073703799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.491513+0000","last_change":"2024-09-12T15:54:18.505427+0000","last_active":"2024-09-12T16:31:50.491513+0000","last_peered":"2024-09-12T16:31:50.491513+0000","last_clean":"2024-09-12T16:31:50.491513+0000","last_became_active":"2024-09-12T15:54:16.503301+0000","last_became_peered":"2024-09-12T15:54:16.503301+0000","last_unstale":"2024-09-12T16:31:50.491513+0000","last_undegraded":"2024-09-12T16:31:50.491513+0000","last_fullsized":"2024-09-12T16:31:50.491513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:26:05.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079584,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.505309+0000","last_change":"2024-09-12T15:54:18.505017+0000","last_active":"2024-09-12T16:33:55.505309+0000","last_peered":"2024-09-12T16:33:55.505309+0000","last_clean":"2024-09-12T16:33:55.505309+0000","last_became_active":"2024-09-12T15:54:16.504202+0000","last_became_peered":"2024-09-12T15:54:16.504202+0000","last_unstale":"2024-09-12T16:33:55.505309+0000","last_undegraded":"2024-09-12T16:33:55.505309+0000","last_fullsized":"2024-09-12T16:33:55.505309+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T02:35:33.901895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007492,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":934,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.363333+0000","last_change":"2024-09-12T15:54:18.508120+0000","last_active":"2024-09-12T16:33:55.363333+0000","last_peered":"2024-09-12T16:33:55.363333+0000","last_clean":"2024-09-12T16:33:55.363333+0000","last_became_active":"2024-09-12T15:54:16.503357+0000","last_became_peered":"2024-09-12T15:54:16.503357+0000","last_unstale":"2024-09-12T16:33:55.363333+0000","last_undegraded":"2024-09-12T16:33:55.363333+0000","last_fullsized":"2024-09-12T16:33:55.363333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:03:29.295197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001241416,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.471249+0000","last_change":"2024-09-12T15:54:18.830314+0000","last_active":"2024-09-12T16:33:55.471249+0000","last_peered":"2024-09-12T16:33:55.471249+0000","last_clean":"2024-09-12T16:33:55.471249+0000","last_became_active":"2024-09-12T15:54:16.505951+0000","last_became_peered":"2024-09-12T15:54:16.505951+0000","last_unstale":"2024-09-12T16:33:55.471249+0000","last_undegraded":"2024-09-12T16:33:55.471249+0000","last_fullsized":"2024-09-12T16:33:55.471249+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:25:36.672104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039973199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:20.461435+0000","last_change":"2024-09-12T15:54:18.830367+0000","last_active":"2024-09-12T16:32:20.461435+0000","last_peered":"2024-09-12T16:32:20.461435+0000","last_clean":"2024-09-12T16:32:20.461435+0000","last_became_active":"2024-09-12T15:54:16.506142+0000","last_became_peered":"2024-09-12T15:54:16.506142+0000","last_unstale":"2024-09-12T16:32:20.461435+0000","last_undegraded":"2024-09-12T16:32:20.461435+0000","last_fullsized":"2024-09-12T16:32:20.461435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:22:17.633227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043801199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.505291+0000","last_change":"2024-09-12T15:54:18.505434+0000","last_active":"2024-09-12T16:33:55.505291+0000","last_peered":"2024-09-12T16:33:55.505291+0000","last_clean":"2024-09-12T16:33:55.505291+0000","last_became_active":"2024-09-12T15:54:16.505397+0000","last_became_peered":"2024-09-12T15:54:16.505397+0000","last_unstale":"2024-09-12T16:33:55.505291+0000","last_undegraded":"2024-09-12T16:33:55.505291+0000","last_fullsized":"2024-09-12T16:33:55.505291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:48:51.220899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099554,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":998,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:15.494402+0000","last_change":"2024-09-12T15:54:11.528229+0000","last_active":"2024-09-12T16:32:15.494402+0000","last_peered":"2024-09-12T16:32:15.494402+0000","last_clean":"2024-09-12T16:32:15.494402+0000","last_became_active":"2024-09-12T15:54:11.527071+0000","last_became_peered":"2024-09-12T15:54:11.527071+0000","last_unstale":"2024-09-12T16:32:15.494402+0000","last_undegraded":"2024-09-12T16:32:15.494402+0000","last_fullsized":"2024-09-12T16:32:15.494402+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_clean_scrub_stamp":"2024-09-12T15:54:10.483728+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:21:13.353299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151436,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220404,"statfs":{"total":96636764160,"available":96481693696,"internally_reserved":0,"allocated":3792896,"data_stored":2029558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125656,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246184,"statfs":{"total":96636764160,"available":96508092416,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179796,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192044,"statfs":{"total":96636764160,"available":96452653056,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:36:00.872 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-12T16:36:00.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T16:36:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T16:36:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T16:36:01.180 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":1714,"stamp":"2024-09-12T16:36:00.770480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456888,"kb_used_data":12280,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658632,"statfs":{"total":289910292480,"available":289442439168,"internally_reserved":0,"allocated":12574720,"data_stored":7269448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":455208271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501538+0000","last_change":"2024-09-12T15:54:18.505765+0000","last_active":"2024-09-12T16:33:20.501538+0000","last_peered":"2024-09-12T16:33:20.501538+0000","last_clean":"2024-09-12T16:33:20.501538+0000","last_became_active":"2024-09-12T15:54:16.504253+0000","last_became_peered":"2024-09-12T15:54:16.504253+0000","last_unstale":"2024-09-12T16:33:20.501538+0000","last_undegraded":"2024-09-12T16:33:20.501538+0000","last_fullsized":"2024-09-12T16:33:20.501538+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:38:53.254216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001172934,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501518+0000","last_change":"2024-09-12T15:54:18.505043+0000","last_active":"2024-09-12T16:33:20.501518+0000","last_peered":"2024-09-12T16:33:20.501518+0000","last_clean":"2024-09-12T16:33:20.501518+0000","last_became_active":"2024-09-12T15:54:16.504145+0000","last_became_peered":"2024-09-12T15:54:16.504145+0000","last_unstale":"2024-09-12T16:33:20.501518+0000","last_undegraded":"2024-09-12T16:33:20.501518+0000","last_fullsized":"2024-09-12T16:33:20.501518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:15:43.919423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073703799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.491513+0000","last_change":"2024-09-12T15:54:18.505427+0000","last_active":"2024-09-12T16:31:50.491513+0000","last_peered":"2024-09-12T16:31:50.491513+0000","last_clean":"2024-09-12T16:31:50.491513+0000","last_became_active":"2024-09-12T15:54:16.503301+0000","last_became_peered":"2024-09-12T15:54:16.503301+0000","last_unstale":"2024-09-12T16:31:50.491513+0000","last_undegraded":"2024-09-12T16:31:50.491513+0000","last_fullsized":"2024-09-12T16:31:50.491513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:26:05.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079584,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.505309+0000","last_change":"2024-09-12T15:54:18.505017+0000","last_active":"2024-09-12T16:33:55.505309+0000","last_peered":"2024-09-12T16:33:55.505309+0000","last_clean":"2024-09-12T16:33:55.505309+0000","last_became_active":"2024-09-12T15:54:16.504202+0000","last_became_peered":"2024-09-12T15:54:16.504202+0000","last_unstale":"2024-09-12T16:33:55.505309+0000","last_undegraded":"2024-09-12T16:33:55.505309+0000","last_fullsized":"2024-09-12T16:33:55.505309+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T02:35:33.901895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007492,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":934,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.363333+0000","last_change":"2024-09-12T15:54:18.508120+0000","last_active":"2024-09-12T16:33:55.363333+0000","last_peered":"2024-09-12T16:33:55.363333+0000","last_clean":"2024-09-12T16:33:55.363333+0000","last_became_active":"2024-09-12T15:54:16.503357+0000","last_became_peered":"2024-09-12T15:54:16.503357+0000","last_unstale":"2024-09-12T16:33:55.363333+0000","last_undegraded":"2024-09-12T16:33:55.363333+0000","last_fullsized":"2024-09-12T16:33:55.363333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:03:29.295197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001241416,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.471249+0000","last_change":"2024-09-12T15:54:18.830314+0000","last_active":"2024-09-12T16:33:55.471249+0000","last_peered":"2024-09-12T16:33:55.471249+0000","last_clean":"2024-09-12T16:33:55.471249+0000","last_became_active":"2024-09-12T15:54:16.505951+0000","last_became_peered":"2024-09-12T15:54:16.505951+0000","last_unstale":"2024-09-12T16:33:55.471249+0000","last_undegraded":"2024-09-12T16:33:55.471249+0000","last_fullsized":"2024-09-12T16:33:55.471249+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:25:36.672104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039973199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:20.461435+0000","last_change":"2024-09-12T15:54:18.830367+0000","last_active":"2024-09-12T16:32:20.461435+0000","last_peered":"2024-09-12T16:32:20.461435+0000","last_clean":"2024-09-12T16:32:20.461435+0000","last_became_active":"2024-09-12T15:54:16.506142+0000","last_became_peered":"2024-09-12T15:54:16.506142+0000","last_unstale":"2024-09-12T16:32:20.461435+0000","last_undegraded":"2024-09-12T16:32:20.461435+0000","last_fullsized":"2024-09-12T16:32:20.461435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:22:17.633227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043801199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.505291+0000","last_change":"2024-09-12T15:54:18.505434+0000","last_active":"2024-09-12T16:33:55.505291+0000","last_peered":"2024-09-12T16:33:55.505291+0000","last_clean":"2024-09-12T16:33:55.505291+0000","last_became_active":"2024-09-12T15:54:16.505397+0000","last_became_peered":"2024-09-12T15:54:16.505397+0000","last_unstale":"2024-09-12T16:33:55.505291+0000","last_undegraded":"2024-09-12T16:33:55.505291+0000","last_fullsized":"2024-09-12T16:33:55.505291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:48:51.220899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099554,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":998,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:15.494402+0000","last_change":"2024-09-12T15:54:11.528229+0000","last_active":"2024-09-12T16:32:15.494402+0000","last_peered":"2024-09-12T16:32:15.494402+0000","last_clean":"2024-09-12T16:32:15.494402+0000","last_became_active":"2024-09-12T15:54:11.527071+0000","last_became_peered":"2024-09-12T15:54:11.527071+0000","last_unstale":"2024-09-12T16:32:15.494402+0000","last_undegraded":"2024-09-12T16:32:15.494402+0000","last_fullsized":"2024-09-12T16:32:15.494402+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_clean_scrub_stamp":"2024-09-12T15:54:10.483728+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:21:13.353299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151436,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220404,"statfs":{"total":96636764160,"available":96481693696,"internally_reserved":0,"allocated":3792896,"data_stored":2029558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125656,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246184,"statfs":{"total":96636764160,"available":96508092416,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179796,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192044,"statfs":{"total":96636764160,"available":96452653056,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:36:01.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-12T16:36:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T16:36:01.478 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":461,"fsid":"b23de1a0-bb6d-4bd1-97d5-57131ff67d17","created":"2024-09-12T15:53:59.551884+0000","modified":"2024-09-12T16:30:45.915531+0000","last_up_change":"2024-09-12T15:54:09.483002+0000","last_in_change":"2024-09-12T15:54:00.421731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-12T15:54:10.147273+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-12T15:54:14.828915+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b2070859-ce30-4b8b-bc50-4d7852c29e68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6801","nonce":196569321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6803","nonce":196569321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6807","nonce":196569321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":196569321},{"type":"v1","addr":"172.21.15.16:6805","nonce":196569321}]},"public_addr":"172.21.15.16:6801/196569321","cluster_addr":"172.21.15.16:6803/196569321","heartbeat_back_addr":"172.21.15.16:6807/196569321","heartbeat_front_addr":"172.21.15.16:6805/196569321","state":["exists","up"]},{"osd":1,"uuid":"6a5979a6-605b-4a99-a7e6-d3950b5e8e86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6817","nonce":2469380610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6819","nonce":2469380610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6823","nonce":2469380610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":2469380610},{"type":"v1","addr":"172.21.15.16:6821","nonce":2469380610}]},"public_addr":"172.21.15.16:6817/2469380610","cluster_addr":"172.21.15.16:6819/2469380610","heartbeat_back_addr":"172.21.15.16:6823/2469380610","heartbeat_front_addr":"172.21.15.16:6821/2469380610","state":["exists","up"]},{"osd":2,"uuid":"4cde2604-d7a6-4c26-8e59-21e0f79c0fa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6809","nonce":791970945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6811","nonce":791970945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6815","nonce":791970945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":791970945},{"type":"v1","addr":"172.21.15.16:6813","nonce":791970945}]},"public_addr":"172.21.15.16:6809/791970945","cluster_addr":"172.21.15.16:6811/791970945","heartbeat_back_addr":"172.21.15.16:6815/791970945","heartbeat_front_addr":"172.21.15.16:6813/791970945","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.498009+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.468066+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-12T15:54:06.772776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-12T16:36:02.492 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-12T16:36:02.492 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-12T16:36:02.632 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T16:36:02.632 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T16:36:02.632 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T16:36:02.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-12T16:36:02.944 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2024-09-12T16:36:02.957 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-12T16:36:02.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-12T16:36:03.082 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T16:36:03.083 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T16:36:03.083 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T16:36:03.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-12T16:36:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2024-09-12T16:36:03.402 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-12T16:36:03.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-12T16:36:03.529 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-12T16:36:03.529 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-12T16:36:03.530 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-12T16:36:03.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-12T16:36:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2024-09-12T16:36:03.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T16:36:04.144 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T16:36:04.144 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T16:36:04.156 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":1715,"stamp":"2024-09-12T16:36:02.770770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456888,"kb_used_data":12280,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658632,"statfs":{"total":289910292480,"available":289442439168,"internally_reserved":0,"allocated":12574720,"data_stored":7269448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":455208271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002593"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501538+0000","last_change":"2024-09-12T15:54:18.505765+0000","last_active":"2024-09-12T16:33:20.501538+0000","last_peered":"2024-09-12T16:33:20.501538+0000","last_clean":"2024-09-12T16:33:20.501538+0000","last_became_active":"2024-09-12T15:54:16.504253+0000","last_became_peered":"2024-09-12T15:54:16.504253+0000","last_unstale":"2024-09-12T16:33:20.501538+0000","last_undegraded":"2024-09-12T16:33:20.501538+0000","last_fullsized":"2024-09-12T16:33:20.501538+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:38:53.254216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001172934,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:20.501518+0000","last_change":"2024-09-12T15:54:18.505043+0000","last_active":"2024-09-12T16:33:20.501518+0000","last_peered":"2024-09-12T16:33:20.501518+0000","last_clean":"2024-09-12T16:33:20.501518+0000","last_became_active":"2024-09-12T15:54:16.504145+0000","last_became_peered":"2024-09-12T15:54:16.504145+0000","last_unstale":"2024-09-12T16:33:20.501518+0000","last_undegraded":"2024-09-12T16:33:20.501518+0000","last_fullsized":"2024-09-12T16:33:20.501518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:15:43.919423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073703799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:31:50.491513+0000","last_change":"2024-09-12T15:54:18.505427+0000","last_active":"2024-09-12T16:31:50.491513+0000","last_peered":"2024-09-12T16:31:50.491513+0000","last_clean":"2024-09-12T16:31:50.491513+0000","last_became_active":"2024-09-12T15:54:16.503301+0000","last_became_peered":"2024-09-12T15:54:16.503301+0000","last_unstale":"2024-09-12T16:31:50.491513+0000","last_undegraded":"2024-09-12T16:31:50.491513+0000","last_fullsized":"2024-09-12T16:31:50.491513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:26:05.342727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079584,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.505309+0000","last_change":"2024-09-12T15:54:18.505017+0000","last_active":"2024-09-12T16:33:55.505309+0000","last_peered":"2024-09-12T16:33:55.505309+0000","last_clean":"2024-09-12T16:33:55.505309+0000","last_became_active":"2024-09-12T15:54:16.504202+0000","last_became_peered":"2024-09-12T15:54:16.504202+0000","last_unstale":"2024-09-12T16:33:55.505309+0000","last_undegraded":"2024-09-12T16:33:55.505309+0000","last_fullsized":"2024-09-12T16:33:55.505309+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T02:35:33.901895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007492,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":934,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.363333+0000","last_change":"2024-09-12T15:54:18.508120+0000","last_active":"2024-09-12T16:33:55.363333+0000","last_peered":"2024-09-12T16:33:55.363333+0000","last_clean":"2024-09-12T16:33:55.363333+0000","last_became_active":"2024-09-12T15:54:16.503357+0000","last_became_peered":"2024-09-12T15:54:16.503357+0000","last_unstale":"2024-09-12T16:33:55.363333+0000","last_undegraded":"2024-09-12T16:33:55.363333+0000","last_fullsized":"2024-09-12T16:33:55.363333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:03:29.295197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001241416,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.471249+0000","last_change":"2024-09-12T15:54:18.830314+0000","last_active":"2024-09-12T16:33:55.471249+0000","last_peered":"2024-09-12T16:33:55.471249+0000","last_clean":"2024-09-12T16:33:55.471249+0000","last_became_active":"2024-09-12T15:54:16.505951+0000","last_became_peered":"2024-09-12T15:54:16.505951+0000","last_unstale":"2024-09-12T16:33:55.471249+0000","last_undegraded":"2024-09-12T16:33:55.471249+0000","last_fullsized":"2024-09-12T16:33:55.471249+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:25:36.672104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039973199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:20.461435+0000","last_change":"2024-09-12T15:54:18.830367+0000","last_active":"2024-09-12T16:32:20.461435+0000","last_peered":"2024-09-12T16:32:20.461435+0000","last_clean":"2024-09-12T16:32:20.461435+0000","last_became_active":"2024-09-12T15:54:16.506142+0000","last_became_peered":"2024-09-12T15:54:16.506142+0000","last_unstale":"2024-09-12T16:32:20.461435+0000","last_undegraded":"2024-09-12T16:32:20.461435+0000","last_fullsized":"2024-09-12T16:32:20.461435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T16:22:17.633227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043801199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:33:55.505291+0000","last_change":"2024-09-12T15:54:18.505434+0000","last_active":"2024-09-12T16:33:55.505291+0000","last_peered":"2024-09-12T16:33:55.505291+0000","last_clean":"2024-09-12T16:33:55.505291+0000","last_became_active":"2024-09-12T15:54:16.505397+0000","last_became_peered":"2024-09-12T15:54:16.505397+0000","last_unstale":"2024-09-12T16:33:55.505291+0000","last_undegraded":"2024-09-12T16:33:55.505291+0000","last_fullsized":"2024-09-12T16:33:55.505291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:15.490308+0000","last_clean_scrub_stamp":"2024-09-12T15:54:15.490308+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T18:48:51.220899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099554,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":998,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:32:15.494402+0000","last_change":"2024-09-12T15:54:11.528229+0000","last_active":"2024-09-12T16:32:15.494402+0000","last_peered":"2024-09-12T16:32:15.494402+0000","last_clean":"2024-09-12T16:32:15.494402+0000","last_became_active":"2024-09-12T15:54:11.527071+0000","last_became_peered":"2024-09-12T15:54:11.527071+0000","last_unstale":"2024-09-12T16:32:15.494402+0000","last_undegraded":"2024-09-12T16:32:15.494402+0000","last_fullsized":"2024-09-12T16:32:15.494402+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T15:54:10.483728+0000","last_clean_scrub_stamp":"2024-09-12T15:54:10.483728+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:21:13.353299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151436,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220404,"statfs":{"total":96636764160,"available":96481693696,"internally_reserved":0,"allocated":3792896,"data_stored":2029558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125656,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246184,"statfs":{"total":96636764160,"available":96508092416,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738874,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179796,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192044,"statfs":{"total":96636764160,"available":96452653056,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-12T15:54:15.490308+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=15, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.160 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-12T15:54:10.483728+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=15, tm_min=54, tm_sec=10, tm_wday=3, tm_yday=256, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=12, tm_hour=16, tm_min=36, tm_sec=1, tm_wday=3, tm_yday=256, tm_isdst=0) 2024-09-12T16:36:04.161 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-12T16:36:24.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-12T16:36:24.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-12T16:36:24.469 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-12T16:36:24.481 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":1725,"stamp":"2024-09-12T16:36:22.774924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456872,"kb_used_data":12280,"kb_used_omap":4,"kb_used_meta":444539,"kb_avail":282658648,"statfs":{"total":289910292480,"available":289442455552,"internally_reserved":0,"allocated":12574720,"data_stored":7269448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":455208271},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":937,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:08.126407+0000","last_change":"2024-09-12T16:36:08.126407+0000","last_active":"2024-09-12T16:36:08.126407+0000","last_peered":"2024-09-12T16:36:08.126407+0000","last_clean":"2024-09-12T16:36:08.126407+0000","last_became_active":"2024-09-12T15:54:16.504253+0000","last_became_peered":"2024-09-12T15:54:16.504253+0000","last_unstale":"2024-09-12T16:36:08.126407+0000","last_undegraded":"2024-09-12T16:36:08.126407+0000","last_fullsized":"2024-09-12T16:36:08.126407+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T16:36:08.126305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T16:36:08.126305+0000","last_clean_scrub_stamp":"2024-09-12T16:36:08.126305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T01:25:44.495725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001172934,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":937,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:09.108882+0000","last_change":"2024-09-12T16:36:09.108882+0000","last_active":"2024-09-12T16:36:09.108882+0000","last_peered":"2024-09-12T16:36:09.108882+0000","last_clean":"2024-09-12T16:36:09.108882+0000","last_became_active":"2024-09-12T15:54:16.504145+0000","last_became_peered":"2024-09-12T15:54:16.504145+0000","last_unstale":"2024-09-12T16:36:09.108882+0000","last_undegraded":"2024-09-12T16:36:09.108882+0000","last_fullsized":"2024-09-12T16:36:09.108882+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T16:36:09.108791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T16:36:09.108791+0000","last_clean_scrub_stamp":"2024-09-12T16:36:09.108791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T23:47:52.376248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073703799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":938,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:07.127678+0000","last_change":"2024-09-12T16:36:07.127678+0000","last_active":"2024-09-12T16:36:07.127678+0000","last_peered":"2024-09-12T16:36:07.127678+0000","last_clean":"2024-09-12T16:36:07.127678+0000","last_became_active":"2024-09-12T15:54:16.503301+0000","last_became_peered":"2024-09-12T15:54:16.503301+0000","last_unstale":"2024-09-12T16:36:07.127678+0000","last_undegraded":"2024-09-12T16:36:07.127678+0000","last_fullsized":"2024-09-12T16:36:07.127678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T16:36:07.127574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T16:36:07.127574+0000","last_clean_scrub_stamp":"2024-09-12T16:36:07.127574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T19:57:39.184623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001079584,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":938,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:05.102187+0000","last_change":"2024-09-12T16:36:05.102187+0000","last_active":"2024-09-12T16:36:05.102187+0000","last_peered":"2024-09-12T16:36:05.102187+0000","last_clean":"2024-09-12T16:36:05.102187+0000","last_became_active":"2024-09-12T15:54:16.504202+0000","last_became_peered":"2024-09-12T15:54:16.504202+0000","last_unstale":"2024-09-12T16:36:05.102187+0000","last_undegraded":"2024-09-12T16:36:05.102187+0000","last_fullsized":"2024-09-12T16:36:05.102187+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T16:36:05.102082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T16:36:05.102082+0000","last_clean_scrub_stamp":"2024-09-12T16:36:05.102082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T21:30:52.660870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007492,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"461'3","reported_seq":943,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:03.511595+0000","last_change":"2024-09-12T16:36:03.511595+0000","last_active":"2024-09-12T16:36:03.511595+0000","last_peered":"2024-09-12T16:36:03.511595+0000","last_clean":"2024-09-12T16:36:03.511595+0000","last_became_active":"2024-09-12T15:54:16.503357+0000","last_became_peered":"2024-09-12T15:54:16.503357+0000","last_unstale":"2024-09-12T16:36:03.511595+0000","last_undegraded":"2024-09-12T16:36:03.511595+0000","last_fullsized":"2024-09-12T16:36:03.511595+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'3","last_scrub_stamp":"2024-09-12T16:36:03.511525+0000","last_deep_scrub":"461'3","last_deep_scrub_stamp":"2024-09-12T16:36:03.511525+0000","last_clean_scrub_stamp":"2024-09-12T16:36:03.511525+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:55:07.795431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001241416,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":940,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:05.196896+0000","last_change":"2024-09-12T16:36:05.196896+0000","last_active":"2024-09-12T16:36:05.196896+0000","last_peered":"2024-09-12T16:36:05.196896+0000","last_clean":"2024-09-12T16:36:05.196896+0000","last_became_active":"2024-09-12T15:54:16.505951+0000","last_became_peered":"2024-09-12T15:54:16.505951+0000","last_unstale":"2024-09-12T16:36:05.196896+0000","last_undegraded":"2024-09-12T16:36:05.196896+0000","last_fullsized":"2024-09-12T16:36:05.196896+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T16:36:05.196799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T16:36:05.196799+0000","last_clean_scrub_stamp":"2024-09-12T16:36:05.196799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T00:34:54.282868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039973199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":941,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:04.159552+0000","last_change":"2024-09-12T16:36:04.159503+0000","last_active":"2024-09-12T16:36:04.159552+0000","last_peered":"2024-09-12T16:36:04.159552+0000","last_clean":"2024-09-12T16:36:04.159552+0000","last_became_active":"2024-09-12T15:54:16.506142+0000","last_became_peered":"2024-09-12T15:54:16.506142+0000","last_unstale":"2024-09-12T16:36:04.159552+0000","last_undegraded":"2024-09-12T16:36:04.159552+0000","last_fullsized":"2024-09-12T16:36:04.159552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-12T16:36:04.159429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-12T16:36:04.159429+0000","last_clean_scrub_stamp":"2024-09-12T16:36:04.159429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T17:40:00.649585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043801199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"461'2","reported_seq":940,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:06.148260+0000","last_change":"2024-09-12T16:36:06.148260+0000","last_active":"2024-09-12T16:36:06.148260+0000","last_peered":"2024-09-12T16:36:06.148260+0000","last_clean":"2024-09-12T16:36:06.148260+0000","last_became_active":"2024-09-12T15:54:16.505397+0000","last_became_peered":"2024-09-12T15:54:16.505397+0000","last_unstale":"2024-09-12T16:36:06.148260+0000","last_undegraded":"2024-09-12T16:36:06.148260+0000","last_fullsized":"2024-09-12T16:36:06.148260+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'2","last_scrub_stamp":"2024-09-12T16:36:06.148171+0000","last_deep_scrub":"461'2","last_deep_scrub_stamp":"2024-09-12T16:36:06.148171+0000","last_clean_scrub_stamp":"2024-09-12T16:36:06.148171+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T20:31:47.248944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001099554,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"461'54","reported_seq":1009,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-12T16:36:04.108113+0000","last_change":"2024-09-12T16:36:04.108048+0000","last_active":"2024-09-12T16:36:04.108113+0000","last_peered":"2024-09-12T16:36:04.108113+0000","last_clean":"2024-09-12T16:36:04.108113+0000","last_became_active":"2024-09-12T15:54:11.527071+0000","last_became_peered":"2024-09-12T15:54:11.527071+0000","last_unstale":"2024-09-12T16:36:04.108113+0000","last_undegraded":"2024-09-12T16:36:04.108113+0000","last_fullsized":"2024-09-12T16:36:04.108113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'54","last_scrub_stamp":"2024-09-12T16:36:04.107979+0000","last_deep_scrub":"461'54","last_deep_scrub_stamp":"2024-09-12T16:36:04.107979+0000","last_clean_scrub_stamp":"2024-09-12T16:36:04.107979+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T22:22:56.895019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151432,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94220408,"statfs":{"total":96636764160,"available":96481697792,"internally_reserved":0,"allocated":3792896,"data_stored":2029558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125648,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":94246192,"statfs":{"total":96636764160,"available":96508100608,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124254661},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738878,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179792,"kb_used_data":4288,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":94192048,"statfs":{"total":96636764160,"available":96452657152,"internally_reserved":0,"allocated":4390912,"data_stored":2619945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179698117},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-12T16:36:24.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-12T16:36:24.801 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph.py", line 1980, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph.py", line 1990, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2867, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2803, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2464, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1696, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1687, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-09-12T16:36:24.801 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-12T16:36:24.801 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-12T16:36:24.802 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-12T16:36:24.802 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:36:24.856 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-12T16:36:24.856 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-12T16:36:24.857 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:36:24.906 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-12T16:36:24.906 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-12T16:36:24.906 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:36:24.954 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-12T16:36:24.954 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-12T16:36:24.954 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-12T16:36:24.954 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:36:24.982 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-12T16:36:24.982 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-12T16:36:24.982 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-12T16:36:24.982 INFO:teuthology.orchestra.run:waiting for 300 2024-09-12T16:36:25.035 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-12T16:36:25.035 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-12T16:36:25.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-12T16:36:25.108 INFO:teuthology.orchestra.run.smithi016.stdout:2024-09-12T15:57:06.603020+0000 mon.a (mon.0) 201 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-12T16:36:25.109 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-12T16:36:25.109 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T16:36:25.109 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-12T16:36:25.426 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T16:36:25.426 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-12T16:36:25.530 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T16:36:25.530 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-12T16:36:25.588 INFO:tasks.ceph:Archiving mon data... 2024-09-12T16:36:25.589 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/data/cluster1.mon.a.tgz 2024-09-12T16:36:25.589 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-12T16:36:25.604 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.0V1QsM7mNN 2024-09-12T16:36:25.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.0V1QsM7mNN 2024-09-12T16:36:25.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.0V1QsM7mNN 2024-09-12T16:36:25.849 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.0V1QsM7mNN is 342KB 2024-09-12T16:36:25.945 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.0V1QsM7mNN 2024-09-12T16:36:25.961 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-12T16:36:25.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T16:36:26.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-12T16:36:26.082 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-12T16:36:26.102 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-12T16:36:26.102 INFO:tasks.ceph:Archiving crash dumps... 2024-09-12T16:36:26.104 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/remote/smithi016/crash 2024-09-12T16:36:26.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-12T16:36:26.148 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/remote/smithi019/crash 2024-09-12T16:36:26.149 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-12T16:36:26.228 INFO:tasks.ceph:Compressing logs... 2024-09-12T16:36:26.228 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T16:36:26.231 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37744.log 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36396.log 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32038.log 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37744.log.gz 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37702.log 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36396.log.gz 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.32038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-12T16:36:26.271 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32038.log.gz 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36374.log 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37702.log.gz 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35883.log 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36060.log 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36374.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36374.log.gz 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.35883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37996.log 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.35883.log.gz 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37954.log 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36060.log.gz 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37786.log 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37996.log.gz 2024-09-12T16:36:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35934.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37954.log.gz 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36267.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37786.log.gz 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.35934.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35934.log.gz 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38122.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36267.log.gz 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27309.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.38122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36140.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38122.log.gz 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.tmp-client.admin.27309.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27309.log.gz 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35858.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36142.log 2024-09-12T16:36:26.273 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36140.log.gz 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.35858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36480.log 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35858.log.gz 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36142.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36142.log.gzgzip 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.32208.log 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36564.log 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36480.log.gz 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.32208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36018.log 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32208.log.gz 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36564.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36564.log.gz 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31800.log 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31998.log 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36018.log.gz 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.31800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31800.log.gz 2024-09-12T16:36:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32137.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.31998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38038.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31998.log.gz 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.32137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37660.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32137.log.gz 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.38038.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38038.log.gz 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31958.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37660.log.gz 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.31958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35802.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31958.log.gz 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.35802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36228.log 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35802.log.gz 2024-09-12T16:36:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36354.log 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37912.log 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36228.log.gz 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37828.log 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36354.log.gz 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37912.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37912.log.gz 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35630.log 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37828.log.gz 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36522.log 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.35630.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35630.log.gz 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/cluster1-client.admin.36141.log 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36522.log.gz 2024-09-12T16:36:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38082.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36141.log.gz 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.38082.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38082.log.gz 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/cluster1-client.admin.37870.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37618.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37870.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35976.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37870.log.gz 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.37618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37618.log.gz 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36268.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.35976.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35976.log.gz/var/log/ceph/cluster1-osd.0.log: 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr: 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-12T16:36:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1-client.admin.36268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36268.log.gz 2024-09-12T16:36:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cluster1.log: 88.2% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-12T16:36:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73802.log 2024-09-12T16:36:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47515.log 2024-09-12T16:36:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58722.log 2024-09-12T16:36:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80560.log 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.58722.log: -- /var/log/ceph/cluster2-client.mirror.48600.log 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66339.log 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47515.log.gz 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80560.log.gz 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58722.log.gz 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62553.log 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66339.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73802.log.gz 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48971.log 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48600.log.gz 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65262.log 2024-09-12T16:36:26.293 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66339.log.gz 2024-09-12T16:36:26.294 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53259.log 2024-09-12T16:36:26.297 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48971.log.gz 2024-09-12T16:36:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61391.log 2024-09-12T16:36:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53259.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65262.log.gz 2024-09-12T16:36:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41489.log 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62553.log.gz 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37624.log 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53259.log.gz 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61391.log.gz 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58888.log 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44084.log 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37624.log.gz 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.72500.log 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41489.log.gz 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58888.log.gz 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39008.log 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44084.log.gz 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.0.72500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53373.log 2024-09-12T16:36:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.39008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51080.log 2024-09-12T16:36:26.299 INFO:teuthology.orchestra.run.smithi019.stderr: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39008.log.gz 2024-09-12T16:36:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/cluster1-client.mirror.53373.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56239.log 2024-09-12T16:36:26.299 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53373.log.gz 2024-09-12T16:36:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40509.log 2024-09-12T16:36:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51229.log 2024-09-12T16:36:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68205.log 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51229.log: gzip 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.72500.log.gz -5 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.67938.log 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40509.log.gz 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51229.log.gz 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64640.log 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68205.log.gz 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56239.log.gz 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.67938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45555.log 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67938.log.gz 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53598.log 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37078.log 2024-09-12T16:36:26.311 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64640.log.gz 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79300.log 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53598.log.gz 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52128.log 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37078.log.gz 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38872.log 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79300.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79300.log.gz 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69510.log 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52128.log.gz 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.38872.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.38872.log.gz 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40888.log 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69533.log 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69510.log.gz 2024-09-12T16:36:26.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66814.log 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr: 82.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40888.log.gz 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.48141.log 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69533.log: /var/log/ceph/cluster2-client.mirror.66814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68835.log 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr: 85.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69533.log.gz 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.1.48141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79972.log 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78460.log 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45555.log.gz 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68835.log.gz 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59931.log 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79972.log.gz 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65076.log 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78460.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78460.log.gz 2024-09-12T16:36:26.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59931.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66814.log.gz 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54266.log 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59298.log 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65076.log.gz 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76443.log 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.48141.log.gz 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43028.log 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54266.log.gz 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59298.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59931.log.gz 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.38217.log 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76443.log.gz 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59298.log.gz 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.43028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59021.log 2024-09-12T16:36:26.314 INFO:teuthology.orchestra.run.smithi019.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43028.log.gz 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.3.38217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57469.log 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55548.log 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57469.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59021.log.gz 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80728.log 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57469.log.gz 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80896.log 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55548.log: /var/log/ceph/cluster2-client.admin.80728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38923.log 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80728.log.gz 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.38217.log.gz 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80896.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80896.log.gz 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66318.log 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38940.log 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38923.log.gz 2024-09-12T16:36:26.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73717.log 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64274.log 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38940.log.gz 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55548.log.gz 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52719.log 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73717.log.gz 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75435.log 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64274.log.gzgzip 2024-09-12T16:36:26.316 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44108.log 2024-09-12T16:36:26.342 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52719.log: /var/log/ceph/cluster2-client.admin.75435.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75435.log.gz 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.38182.log 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66318.log.gz 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43391.log 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44108.log: /var/log/ceph/cluster1-client.mirror.2.38182.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44108.log.gz 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51903.log 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42203.log 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52719.log.gz 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43391.log: /var/log/ceph/cluster1-client.mirror.51903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77913.log 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51903.log.gz 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54630.log 2024-09-12T16:36:26.343 INFO:teuthology.orchestra.run.smithi019.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42203.log.gz 2024-09-12T16:36:26.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48395.log 2024-09-12T16:36:26.344 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77913.log.gz 2024-09-12T16:36:26.344 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57312.log 2024-09-12T16:36:26.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54630.log: /var/log/ceph/cluster2-client.mirror.48395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66546.log 2024-09-12T16:36:26.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57312.log: gzip -5 --verbose -- 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54630.log.gz 2024-09-12T16:36:26.344 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/cluster2-client.mirror.50100.log 2024-09-12T16:36:26.344 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57312.log.gz/var/log/ceph/cluster2-client.mirror.66546.log: 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68588.log 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66546.log.gz 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72919.log 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81148.log 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68588.log.gz 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6%/var/log/ceph/cluster1-client.mirror.72919.log: -- replaced with /var/log/ceph/cluster2-client.mirror.50100.log.gz 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60686.log 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72919.log.gz 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81442.log 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81148.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81148.log.gz 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40398.log 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38735.log 2024-09-12T16:36:26.345 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81442.log.gz 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45403.log 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40398.log.gz 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60139.log 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38735.log: /var/log/ceph/cluster1-client.mirror.45403.log: 68.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38735.log.gz 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60686.log.gz 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77535.log 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66008.log 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60139.log.gz 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77535.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77535.log.gz 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61439.log 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48141.log 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66008.log: /var/log/ceph/cluster1-client.mirror.61439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37120.log 2024-09-12T16:36:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.48141.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61439.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48141.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74517.log 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54447.log 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37120.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66649.log 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54447.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54447.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66008.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66091.log 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45403.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66649.log: gzip -5 91.1% --verbose -- /var/log/ceph/cluster2-client.admin.80098.log 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74517.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66091.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66649.log.gz 2024-09-12T16:36:26.347 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59814.log 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65781.log 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66091.log.gz/var/log/ceph/cluster2-client.admin.80098.log: 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70326.log 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80098.log.gz 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65781.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69600.log 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59814.log.gz 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.70326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63997.log 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65781.log.gz 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74946.log 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69600.log: /var/log/ceph/cluster1-client.mirror.63997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55459.log 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69600.log.gz/var/log/ceph/cluster2-client.mirror.74946.log: 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63997.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74946.log.gz 2024-09-12T16:36:26.348 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53526.log 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49899.log 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55459.log: /var/log/ceph/cluster2-client.mirror.53526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64504.log 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49899.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53526.log.gz 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55459.log.gz 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49899.log.gz 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54427.log 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62782.log 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.54427.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64504.log.gz 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67210.log 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.71758.log 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62782.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54427.log.gz/var/log/ceph/cluster2-client.mirror.67210.log: 2024-09-12T16:36:26.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67210.log.gz 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.81862.log 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.0.71758.log: -- /var/log/ceph/cluster2-client.mirror.72013.log 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81862.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62782.log.gz 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81862.log.gz 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47045.log 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65946.log 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.71758.log.gz 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47045.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47045.log.gz 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63519.log 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48934.log 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63519.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63519.log.gz 2024-09-12T16:36:26.350 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80980.log 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48934.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48934.log.gz 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65946.log.gz 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58909.log 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80980.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80980.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67959.log 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50601.log 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58909.log: /var/log/ceph/cluster1-client.mirror.67959.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67959.log.gz 2024-09-12T16:36:26.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68025.log 2024-09-12T16:36:26.357 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50601.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50601.log.gz 2024-09-12T16:36:26.357 INFO:teuthology.orchestra.run.smithi019.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58909.log.gz 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38650.log 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76863.log 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68025.log.gz 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.38650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72942.log 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76863.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.38650.log.gz 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76863.log.gz 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39669.log 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52084.log 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72942.log: /var/log/ceph/cluster2-client.mirror.39669.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72942.log.gz 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39669.log.gz 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65562.log 2024-09-12T16:36:26.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72211.log 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52084.log: /var/log/ceph/cluster2-client.mirror.65562.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52084.log.gz 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65562.log.gz 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45770.log 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46079.log 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72211.log: /var/log/ceph/cluster2-client.mirror.45770.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72211.log.gz 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50063.log 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46079.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46079.log.gz 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.44420.log 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50063.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45770.log.gz 2024-09-12T16:36:26.359 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63767.log 2024-09-12T16:36:26.360 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.44420.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.44420.log.gz 2024-09-12T16:36:26.360 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76275.log 2024-09-12T16:36:26.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63767.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63767.log.gz 2024-09-12T16:36:26.362 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59138.log 2024-09-12T16:36:26.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76275.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76275.log.gz 2024-09-12T16:36:26.363 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58519.log 2024-09-12T16:36:26.365 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59138.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59138.log.gz 2024-09-12T16:36:26.365 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76485.log 2024-09-12T16:36:26.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58519.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58519.log.gz 2024-09-12T16:36:26.368 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82408.log 2024-09-12T16:36:26.368 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76485.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76485.log.gz 2024-09-12T16:36:26.368 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57678.log 2024-09-12T16:36:26.370 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82408.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82408.log.gz 2024-09-12T16:36:26.370 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78712.log 2024-09-12T16:36:26.374 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57678.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57678.log.gz 2024-09-12T16:36:26.374 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38855.log 2024-09-12T16:36:26.375 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78712.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78712.log.gz 2024-09-12T16:36:26.375 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59346.log 2024-09-12T16:36:26.375 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.38855.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.38855.log.gz 2024-09-12T16:36:26.375 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.61012.log 2024-09-12T16:36:26.376 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59346.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59346.log.gz 2024-09-12T16:36:26.376 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70073.log 2024-09-12T16:36:26.382 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.61012.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.61012.log.gz 2024-09-12T16:36:26.382 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52761.log 2024-09-12T16:36:26.383 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70073.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70073.log.gz 2024-09-12T16:36:26.383 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53076.log 2024-09-12T16:36:26.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52761.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52761.log.gz 2024-09-12T16:36:26.385 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61694.log 2024-09-12T16:36:26.386 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53076.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53076.log.gz 2024-09-12T16:36:26.386 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63743.log 2024-09-12T16:36:26.387 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61694.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61694.log.gz 2024-09-12T16:36:26.388 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81022.log 2024-09-12T16:36:26.389 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63743.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63743.log.gz 2024-09-12T16:36:26.389 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47104.log 2024-09-12T16:36:26.390 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81022.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81022.log.gz 2024-09-12T16:36:26.391 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32790.log 2024-09-12T16:36:26.408 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77031.log 2024-09-12T16:36:26.409 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.32790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32790.log.gz 2024-09-12T16:36:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37960.log 2024-09-12T16:36:26.415 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77031.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77031.log.gz 2024-09-12T16:36:26.417 INFO:teuthology.orchestra.run.smithi019.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.38182.log.gz 2024-09-12T16:36:26.421 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39921.log 2024-09-12T16:36:26.421 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37960.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37960.log.gz 2024-09-12T16:36:26.421 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75393.log 2024-09-12T16:36:26.422 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.39921.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39921.log.gz 2024-09-12T16:36:26.422 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66194.log 2024-09-12T16:36:26.422 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75393.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75393.log.gz 2024-09-12T16:36:26.422 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69105.log 2024-09-12T16:36:26.424 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66194.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66194.log.gz 2024-09-12T16:36:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79258.log 2024-09-12T16:36:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69105.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69105.log.gz 2024-09-12T16:36:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53217.log 2024-09-12T16:36:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79258.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79258.log.gz 2024-09-12T16:36:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68768.log 2024-09-12T16:36:26.429 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53217.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53217.log.gz 2024-09-12T16:36:26.429 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73467.log 2024-09-12T16:36:26.429 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68768.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68768.log.gz 2024-09-12T16:36:26.430 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57876.log 2024-09-12T16:36:26.458 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66587.log 2024-09-12T16:36:26.459 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57876.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57876.log.gz 2024-09-12T16:36:26.460 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.41896.log 2024-09-12T16:36:26.460 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66587.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66587.log.gz 2024-09-12T16:36:26.480 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38336.log 2024-09-12T16:36:26.483 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.2.41896.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.41896.log.gz 2024-09-12T16:36:26.483 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49013.log 2024-09-12T16:36:26.483 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38336.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38336.log.gz 2024-09-12T16:36:26.503 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48395.log.gz 2024-09-12T16:36:26.505 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64890.log 2024-09-12T16:36:26.505 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49013.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49013.log.gz 2024-09-12T16:36:26.505 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58816.log 2024-09-12T16:36:26.506 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64890.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64890.log.gz 2024-09-12T16:36:26.506 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44777.log 2024-09-12T16:36:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58816.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58816.log.gz 2024-09-12T16:36:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61299.log 2024-09-12T16:36:26.510 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44777.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44777.log.gz 2024-09-12T16:36:26.510 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59527.log 2024-09-12T16:36:26.512 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61299.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61299.log.gz 2024-09-12T16:36:26.512 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47082.log 2024-09-12T16:36:26.514 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59527.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59527.log.gz 2024-09-12T16:36:26.514 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73445.log 2024-09-12T16:36:26.514 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47082.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47082.log.gz 2024-09-12T16:36:26.515 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47827.log 2024-09-12T16:36:26.516 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73445.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73445.log.gz 2024-09-12T16:36:26.516 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68903.log 2024-09-12T16:36:26.517 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47827.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47827.log.gz 2024-09-12T16:36:26.518 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32581.log 2024-09-12T16:36:26.519 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68903.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68903.log.gz 2024-09-12T16:36:26.520 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42154.log 2024-09-12T16:36:26.520 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.32581.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32581.log.gz 2024-09-12T16:36:26.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51350.log 2024-09-12T16:36:26.521 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42154.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42154.log.gz 2024-09-12T16:36:26.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75016.log 2024-09-12T16:36:26.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51350.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51350.log.gz 2024-09-12T16:36:26.524 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51251.log 2024-09-12T16:36:26.524 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75016.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75016.log.gz 2024-09-12T16:36:26.524 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55254.log 2024-09-12T16:36:26.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51251.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72013.log.gz 2024-09-12T16:36:26.528 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65678.log 2024-09-12T16:36:26.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55254.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55254.log.gz 2024-09-12T16:36:26.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70232.log 2024-09-12T16:36:26.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65678.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65678.log.gz 2024-09-12T16:36:26.531 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57357.log 2024-09-12T16:36:26.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70232.log: 39.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70232.log.gz 2024-09-12T16:36:26.533 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75645.log 2024-09-12T16:36:26.535 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57357.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57357.log.gz 2024-09-12T16:36:26.535 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69060.log 2024-09-12T16:36:26.539 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75645.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75645.log.gz 2024-09-12T16:36:26.539 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32720.log 2024-09-12T16:36:26.539 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69060.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69060.log.gz 2024-09-12T16:36:26.539 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55986.log 2024-09-12T16:36:26.539 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.32720.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32720.log.gz 2024-09-12T16:36:26.539 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40825.log 2024-09-12T16:36:26.540 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55986.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55986.log.gz 2024-09-12T16:36:26.541 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32541.log 2024-09-12T16:36:26.542 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.40825.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40825.log.gz 2024-09-12T16:36:26.543 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58840.log 2024-09-12T16:36:26.544 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.32541.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32541.log.gz 2024-09-12T16:36:26.544 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77577.log 2024-09-12T16:36:26.545 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58840.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58840.log.gz 2024-09-12T16:36:26.545 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61600.log 2024-09-12T16:36:26.546 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77577.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77577.log.gz 2024-09-12T16:36:26.546 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45426.log 2024-09-12T16:36:26.548 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61600.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61600.log.gz 2024-09-12T16:36:26.548 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50258.log 2024-09-12T16:36:26.552 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45426.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43391.log.gz 2024-09-12T16:36:26.553 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49228.log 2024-09-12T16:36:26.554 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50258.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45426.log.gz 2024-09-12T16:36:26.554 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50258.log.gz 2024-09-12T16:36:26.555 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63287.log 2024-09-12T16:36:26.555 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68655.log 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63287.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49228.log.gz 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45790.log 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63287.log.gz/var/log/ceph/cluster1-client.mirror.68655.log: 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68655.log.gz 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77241.log 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51996.log 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77241.log: /var/log/ceph/cluster1-client.mirror.45790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77241.log.gz 2024-09-12T16:36:26.559 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49727.log 2024-09-12T16:36:26.560 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51996.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51996.log.gz 2024-09-12T16:36:26.562 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50900.log 2024-09-12T16:36:26.562 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49727.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49727.log.gz 2024-09-12T16:36:26.563 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45790.log.gz 2024-09-12T16:36:26.574 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41180.log 2024-09-12T16:36:26.574 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65448.log 2024-09-12T16:36:26.574 INFO:teuthology.orchestra.run.smithi019.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50900.log.gz 2024-09-12T16:36:26.574 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76821.log 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65448.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65448.log.gz 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44180.log 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76821.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76821.log.gz 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47636.log 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44180.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44180.log.gz 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59158.log 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47636.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51080.log.gz 2024-09-12T16:36:26.575 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.39669.log 2024-09-12T16:36:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59158.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41180.log.gz 2024-09-12T16:36:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37201.log 2024-09-12T16:36:26.576 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.39669.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59158.log.gz 2024-09-12T16:36:26.589 INFO:teuthology.orchestra.run.smithi019.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.39669.log.gz 2024-09-12T16:36:26.590 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67187.log 2024-09-12T16:36:26.590 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37201.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69803.log 2024-09-12T16:36:26.590 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.37201.log.gz 2024-09-12T16:36:26.590 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38278.log 2024-09-12T16:36:26.590 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67187.log: /var/log/ceph/cluster1-client.mirror.69803.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69803.log.gz 2024-09-12T16:36:26.590 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67187.log.gz 2024-09-12T16:36:26.591 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48062.log 2024-09-12T16:36:26.591 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48583.log 2024-09-12T16:36:26.591 INFO:teuthology.orchestra.run.smithi019.stderr: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38278.log.gz 2024-09-12T16:36:26.591 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57399.log 2024-09-12T16:36:26.591 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48583.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48583.log.gz 2024-09-12T16:36:26.591 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67334.log 2024-09-12T16:36:26.591 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57399.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48062.log.gz 2024-09-12T16:36:26.592 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68385.log 2024-09-12T16:36:26.592 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67334.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57399.log.gz 2024-09-12T16:36:26.592 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67334.log.gz 2024-09-12T16:36:26.592 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37918.log 2024-09-12T16:36:26.592 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50801.log 2024-09-12T16:36:26.592 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68385.log.gz 2024-09-12T16:36:26.592 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51446.log 2024-09-12T16:36:26.593 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37918.log.gz 2024-09-12T16:36:26.593 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71385.log 2024-09-12T16:36:26.593 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51446.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51446.log.gz 2024-09-12T16:36:26.593 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49564.log 2024-09-12T16:36:26.593 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71385.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71385.log.gz 2024-09-12T16:36:26.593 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67912.log 2024-09-12T16:36:26.597 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49564.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50063.log.gz 2024-09-12T16:36:26.599 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61371.log 2024-09-12T16:36:26.600 INFO:teuthology.orchestra.run.smithi016.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-12T16:36:26.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67912.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67912.log.gz 2024-09-12T16:36:26.602 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62235.log 2024-09-12T16:36:26.609 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61371.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61371.log.gz 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58037.log 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62235.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62235.log.gz 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68070.log 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58037.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58037.log.gz 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77409.log 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68070.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68070.log.gz 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57945.log 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77409.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77409.log.gz 2024-09-12T16:36:26.610 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46869.log 2024-09-12T16:36:26.611 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57945.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57945.log.gz 2024-09-12T16:36:26.612 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44863.log 2024-09-12T16:36:26.612 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46869.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46869.log.gz 2024-09-12T16:36:26.614 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47693.log 2024-09-12T16:36:26.614 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44863.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44863.log.gz 2024-09-12T16:36:26.615 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76947.log 2024-09-12T16:36:26.615 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47693.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47693.log.gz 2024-09-12T16:36:26.616 INFO:teuthology.orchestra.run.smithi016.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-12T16:36:26.618 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65034.log 2024-09-12T16:36:26.618 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76947.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76947.log.gz 2024-09-12T16:36:26.618 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51951.log 2024-09-12T16:36:26.629 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65034.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65034.log.gz 2024-09-12T16:36:26.629 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42110.log 2024-09-12T16:36:26.629 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.51951.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51951.log.gz 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66794.log 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42110.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42110.log.gz 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76401.log 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66794.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66794.log.gz 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70208.log 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76401.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76401.log.gz 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78922.log 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70208.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70208.log.gz 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64869.log 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78922.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78922.log.gz 2024-09-12T16:36:26.630 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67623.log 2024-09-12T16:36:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64869.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64869.log.gz 2024-09-12T16:36:26.633 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67437.log 2024-09-12T16:36:26.633 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67623.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67623.log.gz 2024-09-12T16:36:26.633 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54608.log 2024-09-12T16:36:26.634 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67437.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67437.log.gz 2024-09-12T16:36:26.634 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52947.log 2024-09-12T16:36:26.637 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54608.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54608.log.gz 2024-09-12T16:36:26.637 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48642.log 2024-09-12T16:36:26.638 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52947.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52947.log.gz 2024-09-12T16:36:26.638 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49386.log 2024-09-12T16:36:26.647 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42653.log 2024-09-12T16:36:26.658 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49386.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49386.log.gz 2024-09-12T16:36:26.668 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68745.log 2024-09-12T16:36:26.668 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42653.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42653.log.gz 2024-09-12T16:36:26.680 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74555.log 2024-09-12T16:36:26.680 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68745.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68745.log.gz 2024-09-12T16:36:26.687 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63265.log 2024-09-12T16:36:26.688 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74555.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74555.log.gz 2024-09-12T16:36:26.690 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51251.log.gz 2024-09-12T16:36:26.690 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67809.log 2024-09-12T16:36:26.693 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63265.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63265.log.gz 2024-09-12T16:36:26.694 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78838.log 2024-09-12T16:36:26.694 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67809.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67809.log.gz 2024-09-12T16:36:26.696 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51209.log 2024-09-12T16:36:26.696 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78838.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78838.log.gz 2024-09-12T16:36:26.696 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41115.log 2024-09-12T16:36:26.696 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51209.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51209.log.gz 2024-09-12T16:36:26.698 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57541.log 2024-09-12T16:36:26.698 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41115.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41115.log.gz 2024-09-12T16:36:26.698 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61258.log 2024-09-12T16:36:26.699 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57541.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57541.log.gz 2024-09-12T16:36:26.699 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75519.log 2024-09-12T16:36:26.700 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61258.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61258.log.gz 2024-09-12T16:36:26.700 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64021.log 2024-09-12T16:36:26.701 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75519.log.gz 2024-09-12T16:36:26.701 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57606.log 2024-09-12T16:36:26.702 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64021.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64021.log.gz 2024-09-12T16:36:26.702 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50199.log 2024-09-12T16:36:26.704 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57606.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57606.log.gz 2024-09-12T16:36:26.705 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47498.log 2024-09-12T16:36:26.705 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50199.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50199.log.gz 2024-09-12T16:36:26.706 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58951.log 2024-09-12T16:36:26.706 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47498.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47498.log.gz 2024-09-12T16:36:26.707 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57900.log 2024-09-12T16:36:26.708 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58951.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58951.log.gz 2024-09-12T16:36:26.708 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45340.log 2024-09-12T16:36:26.709 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57900.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57900.log.gz 2024-09-12T16:36:26.709 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69240.log 2024-09-12T16:36:26.715 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45340.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45340.log.gz 2024-09-12T16:36:26.716 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80308.log 2024-09-12T16:36:26.717 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69240.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69240.log.gz 2024-09-12T16:36:26.717 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81106.log 2024-09-12T16:36:26.718 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80308.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80308.log.gz 2024-09-12T16:36:26.718 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71470.log 2024-09-12T16:36:26.719 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81106.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81106.log.gz 2024-09-12T16:36:26.719 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72733.log 2024-09-12T16:36:26.721 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71470.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71470.log.gz 2024-09-12T16:36:26.721 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74706.log 2024-09-12T16:36:26.722 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72733.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72733.log.gz 2024-09-12T16:36:26.722 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48521.log 2024-09-12T16:36:26.723 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74706.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74706.log.gz 2024-09-12T16:36:26.723 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72291.log 2024-09-12T16:36:26.724 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48521.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48521.log.gz 2024-09-12T16:36:26.725 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48415.log 2024-09-12T16:36:26.738 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72291.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47104.log.gz 2024-09-12T16:36:26.740 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52377.log 2024-09-12T16:36:26.740 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48415.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48415.log.gz 2024-09-12T16:36:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54680.log 2024-09-12T16:36:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52377.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52377.log.gz 2024-09-12T16:36:26.742 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60919.log 2024-09-12T16:36:26.743 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54680.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54680.log.gz 2024-09-12T16:36:26.744 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43263.log 2024-09-12T16:36:26.746 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60919.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60919.log.gz 2024-09-12T16:36:26.746 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60368.log 2024-09-12T16:36:26.748 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43263.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43263.log.gz 2024-09-12T16:36:26.748 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78207.log 2024-09-12T16:36:26.748 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60368.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60368.log.gz 2024-09-12T16:36:26.750 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62259.log 2024-09-12T16:36:26.750 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78207.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78207.log.gz 2024-09-12T16:36:26.750 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42292.log 2024-09-12T16:36:26.751 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62259.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62259.log.gz 2024-09-12T16:36:26.751 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66484.log 2024-09-12T16:36:26.758 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42292.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42292.log.gz 2024-09-12T16:36:26.758 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67272.log 2024-09-12T16:36:26.759 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66484.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66484.log.gz 2024-09-12T16:36:26.759 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80854.log 2024-09-12T16:36:26.760 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67272.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67272.log.gz 2024-09-12T16:36:26.762 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74075.log 2024-09-12T16:36:26.762 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80854.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80854.log.gz 2024-09-12T16:36:26.762 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73675.log 2024-09-12T16:36:26.763 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74075.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74075.log.gz 2024-09-12T16:36:26.763 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44244.log 2024-09-12T16:36:26.765 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73675.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73675.log.gz 2024-09-12T16:36:26.765 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52626.log 2024-09-12T16:36:26.771 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44244.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44244.log.gz 2024-09-12T16:36:26.771 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53808.log 2024-09-12T16:36:26.772 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52626.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52626.log.gz 2024-09-12T16:36:26.773 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81568.log 2024-09-12T16:36:26.773 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53808.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53808.log.gz 2024-09-12T16:36:26.775 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80056.log 2024-09-12T16:36:26.775 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81568.log.gz 2024-09-12T16:36:26.775 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74054.log 2024-09-12T16:36:26.776 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80056.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80056.log.gz 2024-09-12T16:36:26.777 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77661.log 2024-09-12T16:36:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74054.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74054.log.gz 2024-09-12T16:36:26.780 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41713.log 2024-09-12T16:36:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77661.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77661.log.gz 2024-09-12T16:36:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65324.log 2024-09-12T16:36:26.787 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41713.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41713.log.gz 2024-09-12T16:36:26.788 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69083.log 2024-09-12T16:36:26.789 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65324.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65324.log.gz 2024-09-12T16:36:26.789 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47330.log 2024-09-12T16:36:26.790 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69083.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69083.log.gz 2024-09-12T16:36:26.790 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45317.log 2024-09-12T16:36:26.801 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82030.log 2024-09-12T16:36:26.812 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45317.log: 93.4% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70326.log.gz 2024-09-12T16:36:26.813 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54543.log 2024-09-12T16:36:26.814 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.45317.log.gz 2024-09-12T16:36:26.814 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82030.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82030.log.gz 2024-09-12T16:36:26.814 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51100.log 2024-09-12T16:36:26.815 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54543.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54543.log.gz 2024-09-12T16:36:26.815 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63953.log 2024-09-12T16:36:26.816 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51100.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51100.log.gz 2024-09-12T16:36:26.816 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77703.log 2024-09-12T16:36:26.818 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63953.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63953.log.gz 2024-09-12T16:36:26.818 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45959.log 2024-09-12T16:36:26.819 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77703.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77703.log.gz 2024-09-12T16:36:26.820 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76737.log 2024-09-12T16:36:26.821 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45959.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45959.log.gz 2024-09-12T16:36:26.821 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49876.log 2024-09-12T16:36:26.823 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76737.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76737.log.gz 2024-09-12T16:36:26.823 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41267.log 2024-09-12T16:36:26.834 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59886.log 2024-09-12T16:36:26.841 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63518.log 2024-09-12T16:36:26.848 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59886.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39772.log 2024-09-12T16:36:26.848 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.63518.log: -- replaced with /var/log/ceph/cluster1-client.mirror.59886.log.gz 2024-09-12T16:36:26.850 INFO:teuthology.orchestra.run.smithi019.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63518.log.gz 2024-09-12T16:36:26.866 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53397.log 2024-09-12T16:36:26.879 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.39772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70304.log 2024-09-12T16:36:26.880 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53397.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53397.log.gz 2024-09-12T16:36:26.894 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66442.log 2024-09-12T16:36:26.895 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.70304.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70304.log.gz 2024-09-12T16:36:26.903 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52224.log 2024-09-12T16:36:26.912 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66442.log: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60455.log 2024-09-12T16:36:26.914 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52224.log: -- replaced with /var/log/ceph/cluster2-client.mirror.73467.log.gz 2024-09-12T16:36:26.914 INFO:teuthology.orchestra.run.smithi019.stderr: 90.3% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52224.log.gz 2024-09-12T16:36:26.914 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52602.log 2024-09-12T16:36:26.915 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60455.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60455.log.gz 2024-09-12T16:36:26.916 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81778.log 2024-09-12T16:36:26.916 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52602.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52602.log.gz 2024-09-12T16:36:26.917 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52536.log 2024-09-12T16:36:26.917 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81778.log.gz 2024-09-12T16:36:26.918 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57965.log 2024-09-12T16:36:26.918 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52536.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52536.log.gz 2024-09-12T16:36:26.919 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44976.log 2024-09-12T16:36:26.921 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57965.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57965.log.gz 2024-09-12T16:36:26.922 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61415.log 2024-09-12T16:36:26.926 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44976.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66442.log.gz 2024-09-12T16:36:26.927 INFO:teuthology.orchestra.run.smithi019.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44976.log.gz 2024-09-12T16:36:26.928 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62970.log 2024-09-12T16:36:26.929 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61415.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61415.log.gz 2024-09-12T16:36:26.929 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60525.log 2024-09-12T16:36:26.930 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62970.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62970.log.gz 2024-09-12T16:36:26.930 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54495.log 2024-09-12T16:36:26.931 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60525.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60525.log.gz 2024-09-12T16:36:26.931 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61012.log 2024-09-12T16:36:26.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54495.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54495.log.gz 2024-09-12T16:36:26.933 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53574.log 2024-09-12T16:36:26.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61012.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61012.log.gz 2024-09-12T16:36:26.933 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75813.log 2024-09-12T16:36:26.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53574.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53574.log.gz 2024-09-12T16:36:26.935 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75183.log 2024-09-12T16:36:26.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75813.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75813.log.gz 2024-09-12T16:36:26.935 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53010.log 2024-09-12T16:36:26.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75183.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75183.log.gz 2024-09-12T16:36:26.937 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38786.log 2024-09-12T16:36:26.938 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53010.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53010.log.gz 2024-09-12T16:36:26.938 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75267.log 2024-09-12T16:36:26.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38786.log: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38786.log.gz 2024-09-12T16:36:26.939 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66112.log 2024-09-12T16:36:26.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75267.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75267.log.gz 2024-09-12T16:36:26.940 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43505.log 2024-09-12T16:36:26.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66112.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66112.log.gz 2024-09-12T16:36:26.941 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65406.log 2024-09-12T16:36:26.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43505.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43505.log.gz 2024-09-12T16:36:26.943 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58240.log 2024-09-12T16:36:26.945 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65406.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65406.log.gz 2024-09-12T16:36:26.946 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.63519.log 2024-09-12T16:36:26.948 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58240.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58240.log.gz 2024-09-12T16:36:26.948 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64952.log 2024-09-12T16:36:26.955 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.2.63519.log: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.63519.log.gz 2024-09-12T16:36:26.955 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58177.log 2024-09-12T16:36:26.957 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64952.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64952.log.gz 2024-09-12T16:36:26.957 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74896.log 2024-09-12T16:36:26.959 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58177.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58177.log.gz 2024-09-12T16:36:26.959 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45211.log 2024-09-12T16:36:26.960 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74896.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74896.log.gz 2024-09-12T16:36:26.960 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69375.log 2024-09-12T16:36:26.966 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45211.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45211.log.gz 2024-09-12T16:36:26.966 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64847.log 2024-09-12T16:36:26.967 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69375.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69375.log.gz 2024-09-12T16:36:26.968 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75981.log 2024-09-12T16:36:26.971 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64847.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64847.log.gz 2024-09-12T16:36:26.971 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75040.log 2024-09-12T16:36:26.971 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75981.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75981.log.gz 2024-09-12T16:36:26.973 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38182.log 2024-09-12T16:36:26.973 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.75040.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75040.log.gz 2024-09-12T16:36:26.974 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40018.log 2024-09-12T16:36:26.986 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67499.log 2024-09-12T16:36:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40018.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40018.log.gz 2024-09-12T16:36:26.989 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47330.log.gz 2024-09-12T16:36:26.990 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74455.log 2024-09-12T16:36:26.990 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67499.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67499.log.gz 2024-09-12T16:36:26.991 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55893.log 2024-09-12T16:36:26.992 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74455.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74455.log.gz 2024-09-12T16:36:27.004 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49406.log 2024-09-12T16:36:27.005 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55893.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55893.log.gz 2024-09-12T16:36:27.016 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47636.log.gz 2024-09-12T16:36:27.017 INFO:teuthology.orchestra.run.smithi019.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38182.log.gz 2024-09-12T16:36:27.018 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71450.log 2024-09-12T16:36:27.018 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49406.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49406.log.gz 2024-09-12T16:36:27.019 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68700.log 2024-09-12T16:36:27.020 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71450.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71450.log.gz 2024-09-12T16:36:27.021 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40798.log 2024-09-12T16:36:27.021 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68700.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68700.log.gz 2024-09-12T16:36:27.022 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55757.log 2024-09-12T16:36:27.023 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.40798.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40798.log.gz 2024-09-12T16:36:27.023 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69150.log 2024-09-12T16:36:27.026 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55757.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55757.log.gz 2024-09-12T16:36:27.026 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79888.log 2024-09-12T16:36:27.027 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69150.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69150.log.gz 2024-09-12T16:36:27.027 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67980.log 2024-09-12T16:36:27.029 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79888.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79888.log.gz 2024-09-12T16:36:27.029 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54290.log 2024-09-12T16:36:27.030 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7%/var/log/ceph/cluster1-client.mirror.67980.log: -- replaced with /var/log/ceph/cluster2-client.mirror.50801.log.gz 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67980.log.gz 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45188.log 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72713.log 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54290.log.gz 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48082.log 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72713.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72713.log.gz 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55736.log 2024-09-12T16:36:27.038 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48082.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48082.log.gz 2024-09-12T16:36:27.039 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55206.log 2024-09-12T16:36:27.039 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55736.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55736.log.gz 2024-09-12T16:36:27.039 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56871.log 2024-09-12T16:36:27.039 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45188.log.gz/var/log/ceph/cluster1-client.mirror.55206.log: 2024-09-12T16:36:27.039 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64573.log 2024-09-12T16:36:27.039 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55206.log.gz 2024-09-12T16:36:27.039 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75561.log 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64573.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56871.log.gz 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82324.log 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75561.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64573.log.gz 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75561.log.gz 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60160.log 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66401.log 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82324.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82324.log.gz 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55162.log 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66401.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66401.log.gz 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55435.log 2024-09-12T16:36:27.043 INFO:teuthology.orchestra.run.smithi019.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60160.log.gz 2024-09-12T16:36:27.047 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67042.log 2024-09-12T16:36:27.047 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55435.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55435.log.gz 2024-09-12T16:36:27.047 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55162.log.gz 2024-09-12T16:36:27.047 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63719.log 2024-09-12T16:36:27.047 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37414.log 2024-09-12T16:36:27.047 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67042.log: /var/log/ceph/cluster2-client.mirror.63719.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67042.log.gz 2024-09-12T16:36:27.048 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63719.log.gz 2024-09-12T16:36:27.048 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40713.log 2024-09-12T16:36:27.048 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.37414.log: -- /var/log/ceph/cluster1-client.mirror.40241.log 2024-09-12T16:36:27.048 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37414.log.gz 2024-09-12T16:36:27.048 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73302.log 2024-09-12T16:36:27.052 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40241.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40713.log.gz 2024-09-12T16:36:27.053 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60939.log 2024-09-12T16:36:27.053 INFO:teuthology.orchestra.run.smithi019.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40241.log.gz 2024-09-12T16:36:27.053 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73302.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73302.log.gz 2024-09-12T16:36:27.054 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60802.log 2024-09-12T16:36:27.054 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50320.log 2024-09-12T16:36:27.055 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60802.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60802.log.gz 2024-09-12T16:36:27.055 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57011.log 2024-09-12T16:36:27.063 INFO:teuthology.orchestra.run.smithi019.stderr: 92.2%/var/log/ceph/cluster2-client.mirror.50320.log: -- replaced with /var/log/ceph/cluster1-client.mirror.60939.log.gz 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50320.log.gz 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62874.log 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73860.log 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57011.log.gz 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78796.log 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73860.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62874.log.gz 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73860.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69173.log 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37328.log 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69173.log: /var/log/ceph/cluster2-client.admin.78796.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78796.log.gz 2024-09-12T16:36:27.064 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69173.log.gz 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38313.log 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49958.log 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38313.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49564.log.gz 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37328.log.gz 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr: 76.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38313.log.gz 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40128.log 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/cluster2-client.mirror.49958.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69398.log 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38259.log 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.69398.log: -- replaced with /var/log/ceph/cluster1-client.mirror.40128.log.gz 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69398.log.gz 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43055.log 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56331.log 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38259.log.gz 2024-09-12T16:36:27.065 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.43055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48432.log 2024-09-12T16:36:27.066 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56331.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43055.log.gz 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73652.log 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56331.log.gz 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48432.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48642.log.gz 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43143.log 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50722.log 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48432.log.gz 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43143.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73652.log.gz 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43143.log.gz 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72963.log 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51681.log 2024-09-12T16:36:27.069 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56628.log 2024-09-12T16:36:27.070 INFO:teuthology.orchestra.run.smithi019.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72963.log.gz 2024-09-12T16:36:27.082 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51681.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72291.log.gz 2024-09-12T16:36:27.082 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54242.log 2024-09-12T16:36:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43817.log 2024-09-12T16:36:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56628.log: /var/log/ceph/cluster1-client.mirror.54242.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56628.log.gz 2024-09-12T16:36:27.085 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54242.log.gz 2024-09-12T16:36:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51789.log 2024-09-12T16:36:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59479.log 2024-09-12T16:36:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51789.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51789.log.gz 2024-09-12T16:36:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71933.log 2024-09-12T16:36:27.087 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59479.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59479.log.gz 2024-09-12T16:36:27.087 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41509.log 2024-09-12T16:36:27.091 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71933.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71933.log.gz 2024-09-12T16:36:27.091 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43817.log.gz 2024-09-12T16:36:27.091 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73425.log 2024-09-12T16:36:27.092 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78081.log 2024-09-12T16:36:27.092 INFO:teuthology.orchestra.run.smithi019.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41509.log.gz 2024-09-12T16:36:27.092 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73425.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73425.log.gz 2024-09-12T16:36:27.095 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60758.log 2024-09-12T16:36:27.095 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.78081.log: /var/log/ceph/cluster2-client.mirror.49817.log 2024-09-12T16:36:27.095 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78081.log.gz 2024-09-12T16:36:27.095 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60091.log 2024-09-12T16:36:27.095 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49817.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49817.log.gz 2024-09-12T16:36:27.096 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45979.log 2024-09-12T16:36:27.096 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60091.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60758.log.gz 2024-09-12T16:36:27.096 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60091.log.gzgzip 2024-09-12T16:36:27.096 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.75064.log 2024-09-12T16:36:27.096 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45979.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45979.log.gz 2024-09-12T16:36:27.098 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72612.log 2024-09-12T16:36:27.098 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75064.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52353.log 2024-09-12T16:36:27.098 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.75064.log.gz 2024-09-12T16:36:27.098 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72612.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72612.log.gz 2024-09-12T16:36:27.100 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45168.log 2024-09-12T16:36:27.100 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.52353.log: /var/log/ceph/cluster2-client.mirror.65843.log 2024-09-12T16:36:27.100 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52353.log.gz 2024-09-12T16:36:27.100 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50522.log 2024-09-12T16:36:27.101 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65843.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65843.log.gz 2024-09-12T16:36:27.101 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41757.log 2024-09-12T16:36:27.102 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50522.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50522.log.gz 2024-09-12T16:36:27.103 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72984.log 2024-09-12T16:36:27.105 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41757.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45168.log.gz 2024-09-12T16:36:27.106 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69960.log 2024-09-12T16:36:27.106 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72984.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72984.log.gz 2024-09-12T16:36:27.107 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40690.log 2024-09-12T16:36:27.107 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69960.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69960.log.gz 2024-09-12T16:36:27.108 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58635.log 2024-09-12T16:36:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40690.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40690.log.gz 2024-09-12T16:36:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71491.log 2024-09-12T16:36:27.115 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58635.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58635.log.gz 2024-09-12T16:36:27.115 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58929.log 2024-09-12T16:36:27.116 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71491.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71491.log.gz 2024-09-12T16:36:27.117 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62346.log 2024-09-12T16:36:27.119 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58929.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58929.log.gz 2024-09-12T16:36:27.119 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55319.log 2024-09-12T16:36:27.120 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62346.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62346.log.gz 2024-09-12T16:36:27.121 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45231.log 2024-09-12T16:36:27.123 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55319.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55319.log.gz 2024-09-12T16:36:27.123 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45297.log 2024-09-12T16:36:27.129 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45231.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45231.log.gz 2024-09-12T16:36:27.129 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55620.log 2024-09-12T16:36:27.135 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45297.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45297.log.gz 2024-09-12T16:36:27.136 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82198.log 2024-09-12T16:36:27.137 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55620.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55620.log.gz 2024-09-12T16:36:27.137 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48025.log 2024-09-12T16:36:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82198.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82198.log.gz 2024-09-12T16:36:27.138 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69915.log 2024-09-12T16:36:27.151 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49270.log 2024-09-12T16:36:27.152 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69915.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69915.log.gz 2024-09-12T16:36:27.158 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74971.log 2024-09-12T16:36:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51429.log 2024-09-12T16:36:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74971.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74971.log.gz 2024-09-12T16:36:27.178 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72898.log 2024-09-12T16:36:27.178 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51429.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51429.log.gz 2024-09-12T16:36:27.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67602.log 2024-09-12T16:36:27.195 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72898.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72898.log.gz 2024-09-12T16:36:27.202 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58772.log 2024-09-12T16:36:27.217 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67602.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67602.log.gz 2024-09-12T16:36:27.221 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48872.log 2024-09-12T16:36:27.223 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58772.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58772.log.gz 2024-09-12T16:36:27.230 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65988.log 2024-09-12T16:36:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65303.log 2024-09-12T16:36:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65988.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48872.log.gz 2024-09-12T16:36:27.244 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65988.log.gz 2024-09-12T16:36:27.252 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58290.log 2024-09-12T16:36:27.253 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65303.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65303.log.gz 2024-09-12T16:36:27.261 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42335.log 2024-09-12T16:36:27.270 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42677.log 2024-09-12T16:36:27.275 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42335.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58290.log.gz 2024-09-12T16:36:27.280 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67871.log 2024-09-12T16:36:27.280 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42677.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42677.log.gz 2024-09-12T16:36:27.288 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42335.log.gz 2024-09-12T16:36:27.289 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38991.log 2024-09-12T16:36:27.292 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67871.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49876.log.gz 2024-09-12T16:36:27.293 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71408.log 2024-09-12T16:36:27.295 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38991.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38991.log.gz 2024-09-12T16:36:27.295 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41581.log 2024-09-12T16:36:27.295 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71408.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71408.log.gz 2024-09-12T16:36:27.296 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62645.log 2024-09-12T16:36:27.297 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41581.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41581.log.gz 2024-09-12T16:36:27.297 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76527.log 2024-09-12T16:36:27.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62645.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62645.log.gz 2024-09-12T16:36:27.299 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56830.log 2024-09-12T16:36:27.299 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76527.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76527.log.gz 2024-09-12T16:36:27.300 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57493.log 2024-09-12T16:36:27.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56830.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56830.log.gz 2024-09-12T16:36:27.302 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67540.log 2024-09-12T16:36:27.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57493.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57493.log.gz 2024-09-12T16:36:27.303 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67871.log.gz 2024-09-12T16:36:27.303 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45360.log 2024-09-12T16:36:27.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67540.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67540.log.gz 2024-09-12T16:36:27.306 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44755.log 2024-09-12T16:36:27.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45360.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45360.log.gz 2024-09-12T16:36:27.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61186.log 2024-09-12T16:36:27.318 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44755.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44755.log.gz 2024-09-12T16:36:27.319 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69353.log 2024-09-12T16:36:27.319 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61186.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61186.log.gz 2024-09-12T16:36:27.319 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59230.log 2024-09-12T16:36:27.320 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69353.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69353.log.gz 2024-09-12T16:36:27.321 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79636.log 2024-09-12T16:36:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59230.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59230.log.gz 2024-09-12T16:36:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37876.log 2024-09-12T16:36:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79636.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79636.log.gz 2024-09-12T16:36:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61958.log 2024-09-12T16:36:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37876.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37876.log.gz 2024-09-12T16:36:27.324 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64972.log 2024-09-12T16:36:27.324 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61958.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61958.log.gz 2024-09-12T16:36:27.325 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62625.log 2024-09-12T16:36:27.328 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64972.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64972.log.gz 2024-09-12T16:36:27.328 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68970.log 2024-09-12T16:36:27.329 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62625.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62625.log.gz 2024-09-12T16:36:27.330 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66711.log 2024-09-12T16:36:27.330 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68970.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68970.log.gz 2024-09-12T16:36:27.331 INFO:teuthology.orchestra.run.smithi016.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-12T16:36:27.331 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73761.log 2024-09-12T16:36:27.332 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66711.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66711.log.gz 2024-09-12T16:36:27.332 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40221.log 2024-09-12T16:36:27.333 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73761.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73761.log.gz 2024-09-12T16:36:27.333 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49601.log 2024-09-12T16:36:27.339 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40221.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40221.log.gz 2024-09-12T16:36:27.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50779.log 2024-09-12T16:36:27.340 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.49601.log: 79.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49601.log.gz 2024-09-12T16:36:27.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57196.log 2024-09-12T16:36:27.341 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50779.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50779.log.gz 2024-09-12T16:36:27.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.73504.log 2024-09-12T16:36:27.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57196.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57196.log.gz 2024-09-12T16:36:27.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37792.log 2024-09-12T16:36:27.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.1.73504.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.73504.log.gz 2024-09-12T16:36:27.347 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57035.log 2024-09-12T16:36:27.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.37792.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37792.log.gz 2024-09-12T16:36:27.348 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49248.log 2024-09-12T16:36:27.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57035.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57035.log.gz 2024-09-12T16:36:27.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65884.log 2024-09-12T16:36:27.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49248.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49248.log.gz 2024-09-12T16:36:27.350 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45489.log 2024-09-12T16:36:27.353 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65884.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65884.log.gz 2024-09-12T16:36:27.354 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52988.log 2024-09-12T16:36:27.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45489.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45489.log.gz 2024-09-12T16:36:27.360 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42725.log 2024-09-12T16:36:27.362 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52988.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52988.log.gz 2024-09-12T16:36:27.362 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53965.log 2024-09-12T16:36:27.363 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42725.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42725.log.gz 2024-09-12T16:36:27.363 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.46486.log 2024-09-12T16:36:27.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53965.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53965.log.gz 2024-09-12T16:36:27.364 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64682.log 2024-09-12T16:36:27.371 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.0.46486.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39772.log.gz 2024-09-12T16:36:27.377 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51681.log.gz 2024-09-12T16:36:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57124.log 2024-09-12T16:36:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70095.log 2024-09-12T16:36:27.377 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57124.log: /var/log/ceph/cluster2-client.mirror.64682.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64682.log.gz 2024-09-12T16:36:27.379 INFO:teuthology.orchestra.run.smithi019.stderr: 92.3% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41267.log.gz 2024-09-12T16:36:27.379 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57124.log.gz 2024-09-12T16:36:27.379 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73782.log 2024-09-12T16:36:27.383 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42179.log 2024-09-12T16:36:27.383 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70095.log.gz 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68093.log 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42179.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73782.log.gz 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75225.log 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42179.log.gz 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.47330.log 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68093.log: /var/log/ceph/cluster2-client.admin.75225.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75225.log.gz 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68093.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71954.log 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.3.47330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55570.log 2024-09-12T16:36:27.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71954.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71954.log.gz 2024-09-12T16:36:27.385 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81190.log 2024-09-12T16:36:27.385 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55570.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55570.log.gz 2024-09-12T16:36:27.385 INFO:teuthology.orchestra.run.smithi019.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.46486.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52425.log 2024-09-12T16:36:27.386 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:27.386 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81190.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81190.log.gz 2024-09-12T16:36:27.386 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61670.log 2024-09-12T16:36:27.387 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52425.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52425.log.gz 2024-09-12T16:36:27.389 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63815.log 2024-09-12T16:36:27.389 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52287.log 2024-09-12T16:36:27.389 INFO:teuthology.orchestra.run.smithi019.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61670.log.gz 2024-09-12T16:36:27.389 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73325.log 2024-09-12T16:36:27.389 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52287.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63815.log.gz 2024-09-12T16:36:27.393 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66670.log 2024-09-12T16:36:27.393 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73325.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52287.log.gz 2024-09-12T16:36:27.393 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73325.log.gz 2024-09-12T16:36:27.393 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42701.log 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71571.log 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66670.log.gz 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49505.log 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71571.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42701.log.gz 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49542.log 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71571.log.gz 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.67099.log 2024-09-12T16:36:27.394 INFO:teuthology.orchestra.run.smithi019.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49505.log.gz 2024-09-12T16:36:27.397 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49542.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49542.log.gz 2024-09-12T16:36:27.397 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75855.log 2024-09-12T16:36:27.397 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.1.67099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74476.log 2024-09-12T16:36:27.397 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75855.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75855.log.gz 2024-09-12T16:36:27.398 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46320.log 2024-09-12T16:36:27.398 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74476.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74476.log.gz 2024-09-12T16:36:27.398 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64182.log 2024-09-12T16:36:27.400 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46320.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46320.log.gz 2024-09-12T16:36:27.401 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54772.log 2024-09-12T16:36:27.403 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64182.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64182.log.gz 2024-09-12T16:36:27.403 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60645.log 2024-09-12T16:36:27.404 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54772.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54772.log.gz 2024-09-12T16:36:27.404 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65427.log 2024-09-12T16:36:27.405 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60645.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60645.log.gz 2024-09-12T16:36:27.406 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55001.log 2024-09-12T16:36:27.407 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65427.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65427.log.gz 2024-09-12T16:36:27.408 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49749.log 2024-09-12T16:36:27.408 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55001.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55001.log.gz 2024-09-12T16:36:27.409 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46812.log 2024-09-12T16:36:27.412 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49749.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.67099.log.gz 2024-09-12T16:36:27.413 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50959.log 2024-09-12T16:36:27.433 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60344.log 2024-09-12T16:36:27.450 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74748.log 2024-09-12T16:36:27.457 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60344.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60344.log.gz 2024-09-12T16:36:27.457 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61487.log 2024-09-12T16:36:27.471 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74748.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74748.log.gz 2024-09-12T16:36:27.476 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67375.log 2024-09-12T16:36:27.476 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61487.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61487.log.gz 2024-09-12T16:36:27.485 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67354.log 2024-09-12T16:36:27.485 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67375.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67375.log.gz 2024-09-12T16:36:27.494 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52473.log 2024-09-12T16:36:27.494 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67354.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50722.log.gz 2024-09-12T16:36:27.495 INFO:teuthology.orchestra.run.smithi019.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67354.log.gz 2024-09-12T16:36:27.504 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49958.log.gz 2024-09-12T16:36:27.505 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54700.log 2024-09-12T16:36:27.506 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52473.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52473.log.gz 2024-09-12T16:36:27.508 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42014.log 2024-09-12T16:36:27.508 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54700.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54700.log.gz 2024-09-12T16:36:27.508 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47240.log 2024-09-12T16:36:27.509 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.42014.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42014.log.gz 2024-09-12T16:36:27.510 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79006.log 2024-09-12T16:36:27.511 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47240.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47240.log.gz 2024-09-12T16:36:27.511 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72084.log 2024-09-12T16:36:27.513 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79006.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79006.log.gz 2024-09-12T16:36:27.513 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82072.log 2024-09-12T16:36:27.513 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72084.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72084.log.gz 2024-09-12T16:36:27.514 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41650.log 2024-09-12T16:36:27.515 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82072.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82072.log.gz 2024-09-12T16:36:27.515 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38667.log 2024-09-12T16:36:27.520 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41650.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41650.log.gz 2024-09-12T16:36:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66773.log 2024-09-12T16:36:27.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.38667.log: 73.7% -- replaced with /var/log/ceph/cluster1-client.admin.38667.log.gz 2024-09-12T16:36:27.523 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68340.log 2024-09-12T16:36:27.524 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66773.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66773.log.gz 2024-09-12T16:36:27.524 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71508.log 2024-09-12T16:36:27.525 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68340.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68340.log.gz 2024-09-12T16:36:27.525 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49369.log 2024-09-12T16:36:27.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71508.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71508.log.gz 2024-09-12T16:36:27.527 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37540.log 2024-09-12T16:36:27.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49369.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49369.log.gz 2024-09-12T16:36:27.528 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40670.log 2024-09-12T16:36:27.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37540.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37540.log.gz 2024-09-12T16:36:27.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80434.log 2024-09-12T16:36:27.536 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40670.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40670.log.gz 2024-09-12T16:36:27.536 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51159.log 2024-09-12T16:36:27.537 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80434.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80434.log.gz 2024-09-12T16:36:27.537 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.57787.log 2024-09-12T16:36:27.553 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46770.log 2024-09-12T16:36:27.568 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.1.57787.log: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66897.log 2024-09-12T16:36:27.568 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.1.57787.log.gz 2024-09-12T16:36:27.581 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46770.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46770.log.gz 2024-09-12T16:36:27.588 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40529.log 2024-09-12T16:36:27.589 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66897.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66897.log.gz 2024-09-12T16:36:27.590 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49270.log.gz 2024-09-12T16:36:27.593 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32408.log 2024-09-12T16:36:27.593 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40529.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40529.log.gz 2024-09-12T16:36:27.594 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58680.log 2024-09-12T16:36:27.594 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.32408.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32408.log.gz 2024-09-12T16:36:27.594 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38044.log 2024-09-12T16:36:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58680.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58680.log.gz 2024-09-12T16:36:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64660.log 2024-09-12T16:36:27.597 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.38044.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38044.log.gz 2024-09-12T16:36:27.598 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79174.log 2024-09-12T16:36:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64660.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64660.log.gz 2024-09-12T16:36:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59274.log 2024-09-12T16:36:27.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79174.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79174.log.gz 2024-09-12T16:36:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82366.log 2024-09-12T16:36:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59274.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59274.log.gz 2024-09-12T16:36:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73042.log 2024-09-12T16:36:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82366.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82366.log.gz 2024-09-12T16:36:27.604 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60898.log 2024-09-12T16:36:27.606 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73042.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73042.log.gz 2024-09-12T16:36:27.606 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72126.log 2024-09-12T16:36:27.608 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60898.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60898.log.gz 2024-09-12T16:36:27.608 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56078.log 2024-09-12T16:36:27.609 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72126.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72126.log.gz 2024-09-12T16:36:27.609 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52333.log 2024-09-12T16:36:27.611 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56078.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56078.log.gz 2024-09-12T16:36:27.611 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77787.log 2024-09-12T16:36:27.614 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52333.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52333.log.gz 2024-09-12T16:36:27.614 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44286.log 2024-09-12T16:36:27.614 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77787.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77787.log.gz 2024-09-12T16:36:27.614 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68003.log 2024-09-12T16:36:27.616 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44286.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44286.log.gz 2024-09-12T16:36:27.616 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41341.log 2024-09-12T16:36:27.617 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68003.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68003.log.gz 2024-09-12T16:36:27.617 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38278.log 2024-09-12T16:36:27.619 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41341.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49749.log.gz 2024-09-12T16:36:27.619 INFO:teuthology.orchestra.run.smithi019.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41341.log.gz 2024-09-12T16:36:27.619 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46753.log 2024-09-12T16:36:27.620 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.1.38278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-12T16:36:27.621 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46753.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46753.log.gz 2024-09-12T16:36:27.622 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41203.log 2024-09-12T16:36:27.622 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-osd.2.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38278.log.gz 2024-09-12T16:36:27.622 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53871.log 2024-09-12T16:36:27.628 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41203.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41203.log.gz 2024-09-12T16:36:27.628 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66504.log 2024-09-12T16:36:27.629 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53871.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53871.log.gz 2024-09-12T16:36:27.630 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.74342.log 2024-09-12T16:36:27.633 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66504.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66504.log.gz 2024-09-12T16:36:27.633 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71550.log 2024-09-12T16:36:27.635 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.0.74342.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.74342.log.gz 2024-09-12T16:36:27.635 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72249.log 2024-09-12T16:36:27.648 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74768.log 2024-09-12T16:36:27.649 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72249.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72249.log.gz 2024-09-12T16:36:27.655 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41757.log.gz 2024-09-12T16:36:27.656 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67313.log 2024-09-12T16:36:27.659 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74768.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74768.log.gz 2024-09-12T16:36:27.660 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37327.log 2024-09-12T16:36:27.662 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67313.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67313.log.gz 2024-09-12T16:36:27.662 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39992.log 2024-09-12T16:36:27.662 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37327.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37327.log.gz 2024-09-12T16:36:27.663 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54174.log 2024-09-12T16:36:27.664 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.39992.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.39992.log.gz 2024-09-12T16:36:27.664 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63839.log 2024-09-12T16:36:27.666 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54174.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54174.log.gz 2024-09-12T16:36:27.666 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46407.log 2024-09-12T16:36:27.667 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63839.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63839.log.gz 2024-09-12T16:36:27.668 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44330.log 2024-09-12T16:36:27.669 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46407.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46407.log.gz 2024-09-12T16:36:27.669 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73025.log 2024-09-12T16:36:27.670 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.44330.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.44330.log.gz 2024-09-12T16:36:27.671 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48045.log 2024-09-12T16:36:27.671 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73025.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73025.log.gz 2024-09-12T16:36:27.673 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47557.log 2024-09-12T16:36:27.673 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48045.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48045.log.gz 2024-09-12T16:36:27.674 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40777.log 2024-09-12T16:36:27.680 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47557.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48025.log.gz 2024-09-12T16:36:27.681 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78375.log 2024-09-12T16:36:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62946.log 2024-09-12T16:36:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78375.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78375.log.gz 2024-09-12T16:36:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45856.log 2024-09-12T16:36:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62946.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62946.log.gz 2024-09-12T16:36:27.721 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79930.log 2024-09-12T16:36:27.726 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45856.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45856.log.gz 2024-09-12T16:36:27.740 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68925.log 2024-09-12T16:36:27.740 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79930.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79930.log.gz 2024-09-12T16:36:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59387.log 2024-09-12T16:36:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68925.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68925.log.gz 2024-09-12T16:36:27.775 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56652.log 2024-09-12T16:36:27.777 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59387.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59387.log.gz 2024-09-12T16:36:27.793 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40625.log 2024-09-12T16:36:27.793 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56652.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56652.log.gz 2024-09-12T16:36:27.805 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81652.log 2024-09-12T16:36:27.806 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40625.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40625.log.gz 2024-09-12T16:36:27.822 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80518.log 2024-09-12T16:36:27.822 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81652.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81652.log.gz 2024-09-12T16:36:27.831 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63152.log 2024-09-12T16:36:27.831 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80518.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80518.log.gz 2024-09-12T16:36:27.850 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45833.log 2024-09-12T16:36:27.850 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63152.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63152.log.gz 2024-09-12T16:36:27.858 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37202.log 2024-09-12T16:36:27.864 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45833.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45833.log.gz 2024-09-12T16:36:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45575.log 2024-09-12T16:36:27.867 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37202.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37202.log.gz 2024-09-12T16:36:27.877 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53305.log 2024-09-12T16:36:27.886 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45575.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.47330.log.gz 2024-09-12T16:36:27.886 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65967.log 2024-09-12T16:36:27.887 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53305.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50959.log.gz 2024-09-12T16:36:27.906 INFO:teuthology.orchestra.run.smithi019.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53305.log.gz 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51159.log.gz 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76779.log 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.65967.log: /var/log/ceph/cluster1-client.mirror.43239.log 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71550.log.gz 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76779.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76779.log.gz 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65967.log.gz 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48474.log 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52401.log 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48474.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43239.log.gz 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77325.log 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44692.log 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4%/var/log/ceph/cluster2-client.admin.77325.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45575.log.gz/var/log/ceph/cluster1-client.mirror.52401.log: 2024-09-12T16:36:27.907 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77325.log.gz 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52401.log.gz 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60392.log 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/cluster2-client.mirror.44692.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55049.log 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81820.log 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60392.log.gz/var/log/ceph/cluster1-client.mirror.55049.log: 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55049.log.gz 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69465.log 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68048.log 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81820.log.gz 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81064.log 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68048.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69465.log.gz 2024-09-12T16:36:27.908 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68048.log.gz 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69195.log 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81064.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51745.log 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.81064.log.gz 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38906.log 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.51745.log: 1.3% 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69195.log.gz 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51745.log.gz 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75729.log 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44692.log.gz/var/log/ceph/cluster2-client.mirror.38906.log: 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr: 68.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38906.log.gz 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79426.log 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75729.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75729.log.gz 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.62922.log 2024-09-12T16:36:27.909 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79426.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44383.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.79426.log.gz 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/cluster1-client.mirror.62922.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53196.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50858.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62922.log.gz 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44383.log: /var/log/ceph/cluster1-client.mirror.53196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52152.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50858.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53196.log.gz 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50858.log.gz 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59838.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47594.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44383.log.gz 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52152.log.gz 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62512.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.47594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48620.log 2024-09-12T16:36:27.910 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59838.log.gz 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48662.log 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62512.log: /var/log/ceph/cluster2-client.mirror.48620.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47594.log.gz 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48620.log.gz 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62512.log.gz 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55112.log 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45727.log 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48662.log.gz 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51766.log 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57146.log 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55112.log.gz 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.51766.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51766.log.gz 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67000.log 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71913.log 2024-09-12T16:36:27.911 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67000.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57146.log.gz 2024-09-12T16:36:27.919 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54194.log 2024-09-12T16:36:27.919 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71913.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71913.log.gz 2024-09-12T16:36:27.919 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40308.log 2024-09-12T16:36:27.919 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54194.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54194.log.gz 2024-09-12T16:36:27.919 INFO:teuthology.orchestra.run.smithi019.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67000.log.gz 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46449.log 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.40308.log: 93.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.45727.log.gz -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42963.log 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40308.log.gz 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.46449.log: -- /var/log/ceph/cluster2-client.mirror.50680.log 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80392.log 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50680.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50680.log.gz 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62416.log 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80392.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80392.log.gz 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81400.log 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62416.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62416.log.gz 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62187.log 2024-09-12T16:36:27.920 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81400.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81400.log.gz 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49707.log 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62187.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62187.log.gz 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79678.log 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49707.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42963.log.gz 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63880.log 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79678.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79678.log.gz 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49707.log.gz 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.39809.log 2024-09-12T16:36:27.921 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.63880.log: -- /var/log/ceph/cluster2-client.admin.79342.log 2024-09-12T16:36:27.923 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.1.39809.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63880.log.gz 2024-09-12T16:36:27.923 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.72394.log 2024-09-12T16:36:27.923 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79342.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79342.log.gz 2024-09-12T16:36:27.923 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44308.log 2024-09-12T16:36:27.926 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.72394.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46812.log.gz 2024-09-12T16:36:27.926 INFO:teuthology.orchestra.run.smithi019.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.72394.log.gz 2024-09-12T16:36:27.927 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63626.log 2024-09-12T16:36:27.927 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50917.log 2024-09-12T16:36:27.928 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63626.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47557.log.gz 2024-09-12T16:36:27.929 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47436.log 2024-09-12T16:36:27.929 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50917.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63626.log.gz 2024-09-12T16:36:27.930 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79552.log 2024-09-12T16:36:27.931 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50917.log.gz 2024-09-12T16:36:27.931 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66463.log 2024-09-12T16:36:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79552.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79552.log.gz 2024-09-12T16:36:27.932 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47436.log.gz 2024-09-12T16:36:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62006.log 2024-09-12T16:36:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55528.log 2024-09-12T16:36:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62006.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66463.log.gz 2024-09-12T16:36:27.934 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62006.log.gz 2024-09-12T16:36:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77157.log 2024-09-12T16:36:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81274.log 2024-09-12T16:36:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77157.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77157.log.gz 2024-09-12T16:36:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.59592.log 2024-09-12T16:36:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81274.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81274.log.gz 2024-09-12T16:36:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75477.log 2024-09-12T16:36:27.936 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55528.log.gz 2024-09-12T16:36:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.3.59592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68858.log 2024-09-12T16:36:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75477.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75477.log.gz 2024-09-12T16:36:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38633.log 2024-09-12T16:36:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68858.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68858.log.gz 2024-09-12T16:36:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65616.log 2024-09-12T16:36:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.38633.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.38633.log.gz 2024-09-12T16:36:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80182.log 2024-09-12T16:36:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65616.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65616.log.gz 2024-09-12T16:36:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44647.log 2024-09-12T16:36:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80182.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80182.log.gz 2024-09-12T16:36:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47887.log 2024-09-12T16:36:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44647.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44647.log.gz 2024-09-12T16:36:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55640.log 2024-09-12T16:36:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47887.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47887.log.gz 2024-09-12T16:36:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46199.log 2024-09-12T16:36:27.945 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55640.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55640.log.gz 2024-09-12T16:36:27.945 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64806.log 2024-09-12T16:36:27.947 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46199.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46199.log.gz 2024-09-12T16:36:27.947 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41605.log 2024-09-12T16:36:27.949 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64806.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64806.log.gz 2024-09-12T16:36:27.949 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56447.log 2024-09-12T16:36:27.949 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41605.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.39809.log.gz 2024-09-12T16:36:27.951 INFO:teuthology.orchestra.run.smithi019.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41605.log.gz 2024-09-12T16:36:27.951 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50838.log 2024-09-12T16:36:27.951 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61739.log 2024-09-12T16:36:27.951 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56447.log.gz 2024-09-12T16:36:27.952 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52923.log 2024-09-12T16:36:27.952 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61739.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50838.log.gz 2024-09-12T16:36:27.952 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43880.log 2024-09-12T16:36:27.953 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52923.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52923.log.gz 2024-09-12T16:36:27.953 INFO:teuthology.orchestra.run.smithi019.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61739.log.gz 2024-09-12T16:36:27.953 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69330.log 2024-09-12T16:36:27.954 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51855.log 2024-09-12T16:36:27.954 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43880.log: /var/log/ceph/cluster1-client.mirror.69330.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69330.log.gz 2024-09-12T16:36:27.955 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55917.log 2024-09-12T16:36:27.955 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51855.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51855.log.gz 2024-09-12T16:36:27.956 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49834.log 2024-09-12T16:36:27.958 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55917.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55917.log.gz 2024-09-12T16:36:27.958 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75771.log 2024-09-12T16:36:27.958 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49834.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49834.log.gz 2024-09-12T16:36:27.960 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77829.log 2024-09-12T16:36:27.960 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75771.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75771.log.gz 2024-09-12T16:36:27.960 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36862.log 2024-09-12T16:36:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77829.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43880.log.gz 2024-09-12T16:36:27.961 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77829.log.gz 2024-09-12T16:36:27.962 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61279.log 2024-09-12T16:36:27.962 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.36862.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46938.log 2024-09-12T16:36:27.963 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36862.log.gz 2024-09-12T16:36:27.963 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59459.log 2024-09-12T16:36:27.963 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46938.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46938.log.gz 2024-09-12T16:36:27.964 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51879.log 2024-09-12T16:36:27.964 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59459.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61279.log.gz 2024-09-12T16:36:27.965 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59459.log.gz 2024-09-12T16:36:27.965 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42749.log 2024-09-12T16:36:27.965 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.51879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76107.log 2024-09-12T16:36:27.965 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51879.log.gz 2024-09-12T16:36:27.966 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59722.log 2024-09-12T16:36:27.966 INFO:teuthology.orchestra.run.smithi019.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42749.log.gz 2024-09-12T16:36:27.966 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76107.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76107.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56006.log 2024-09-12T16:36:27.967 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:27.967 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69825.log 2024-09-12T16:36:27.969 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56006.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59722.log.gz 2024-09-12T16:36:27.971 INFO:teuthology.orchestra.run.smithi019.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56006.log.gz 2024-09-12T16:36:27.971 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47849.log 2024-09-12T16:36:27.971 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37708.log 2024-09-12T16:36:27.971 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69825.log.gz 2024-09-12T16:36:27.971 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81736.log 2024-09-12T16:36:27.971 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.37708.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37708.log.gz 2024-09-12T16:36:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43123.log 2024-09-12T16:36:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81736.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81736.log.gz 2024-09-12T16:36:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60505.log 2024-09-12T16:36:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43123.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43123.log.gz 2024-09-12T16:36:27.974 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66608.log 2024-09-12T16:36:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60505.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60505.log.gz 2024-09-12T16:36:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61781.log 2024-09-12T16:36:27.977 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66608.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66608.log.gz 2024-09-12T16:36:27.977 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45085.log 2024-09-12T16:36:27.979 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61781.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61781.log.gz 2024-09-12T16:36:27.980 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.56717.log 2024-09-12T16:36:27.980 INFO:teuthology.orchestra.run.smithi019.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.59592.log.gz 2024-09-12T16:36:27.980 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62898.log 2024-09-12T16:36:27.982 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.3.56717.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.56717.log.gz 2024-09-12T16:36:27.983 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81484.log 2024-09-12T16:36:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62898.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62898.log.gz 2024-09-12T16:36:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57240.log 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81484.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81484.log.gz 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45085.log.gz 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/cluster2-client.admin.37434.log 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/cluster1-client.mirror.57240.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.40888.log 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57240.log.gz 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37434.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37434.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45704.log 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.0.40888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64456.log 2024-09-12T16:36:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45704.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45704.log.gz 2024-09-12T16:36:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61162.log 2024-09-12T16:36:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64456.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64456.log.gz 2024-09-12T16:36:27.997 INFO:teuthology.orchestra.run.smithi019.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.40888.log.gz 2024-09-12T16:36:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77367.log 2024-09-12T16:36:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54124.log 2024-09-12T16:36:27.998 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61162.log.gz 2024-09-12T16:36:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77367.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77367.log.gz 2024-09-12T16:36:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45641.log 2024-09-12T16:36:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66422.log 2024-09-12T16:36:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45641.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54124.log.gz 2024-09-12T16:36:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54883.log 2024-09-12T16:36:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66422.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66422.log.gz 2024-09-12T16:36:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57264.log 2024-09-12T16:36:28.001 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54883.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54883.log.gz 2024-09-12T16:36:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46891.log 2024-09-12T16:36:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57264.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57264.log.gz 2024-09-12T16:36:28.003 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81232.log 2024-09-12T16:36:28.004 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46891.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45641.log.gz 2024-09-12T16:36:28.004 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71971.log 2024-09-12T16:36:28.005 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81232.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81232.log.gz 2024-09-12T16:36:28.005 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50004.log 2024-09-12T16:36:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71971.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71971.log.gz 2024-09-12T16:36:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42251.log 2024-09-12T16:36:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50004.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50004.log.gz 2024-09-12T16:36:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60433.log 2024-09-12T16:36:28.009 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42251.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42251.log.gz 2024-09-12T16:36:28.009 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58659.log 2024-09-12T16:36:28.011 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60433.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60433.log.gz 2024-09-12T16:36:28.012 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58382.log 2024-09-12T16:36:28.012 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58659.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58659.log.gz 2024-09-12T16:36:28.013 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67099.log 2024-09-12T16:36:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58382.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58382.log.gz 2024-09-12T16:36:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54082.log 2024-09-12T16:36:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.67099.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67099.log.gz 2024-09-12T16:36:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45469.log 2024-09-12T16:36:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54082.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54082.log.gz 2024-09-12T16:36:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72755.log 2024-09-12T16:36:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45469.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45469.log.gz 2024-09-12T16:36:28.024 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49522.log 2024-09-12T16:36:28.041 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42773.log 2024-09-12T16:36:28.042 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49522.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49522.log.gz 2024-09-12T16:36:28.047 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48474.log.gz 2024-09-12T16:36:28.048 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48993.log 2024-09-12T16:36:28.048 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42773.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42773.log.gz 2024-09-12T16:36:28.049 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56423.log 2024-09-12T16:36:28.062 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76065.log 2024-09-12T16:36:28.063 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56423.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56423.log.gz 2024-09-12T16:36:28.077 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-12T16:36:28.077 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76065.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76065.log.gz 2024-09-12T16:36:28.086 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40777.log.gz 2024-09-12T16:36:28.087 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38684.log 2024-09-12T16:36:28.092 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2.audit.log: 92.5% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-12T16:36:28.092 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40999.log 2024-09-12T16:36:28.094 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38684.log: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38684.log.gz 2024-09-12T16:36:28.094 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74846.log 2024-09-12T16:36:28.095 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40999.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40999.log.gz 2024-09-12T16:36:28.096 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52582.log 2024-09-12T16:36:28.096 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74846.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74846.log.gz 2024-09-12T16:36:28.097 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45876.log 2024-09-12T16:36:28.098 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52582.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52582.log.gz 2024-09-12T16:36:28.099 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59093.log 2024-09-12T16:36:28.105 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45876.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45876.log.gz 2024-09-12T16:36:28.105 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52851.log 2024-09-12T16:36:28.106 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59093.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59093.log.gz 2024-09-12T16:36:28.107 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47478.log 2024-09-12T16:36:28.107 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52851.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52851.log.gz 2024-09-12T16:36:28.108 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54724.log 2024-09-12T16:36:28.119 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.40398.log 2024-09-12T16:36:28.120 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54724.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54724.log.gz 2024-09-12T16:36:28.133 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56717.log 2024-09-12T16:36:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.3.40398.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.40398.log.gz 2024-09-12T16:36:28.142 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70249.log 2024-09-12T16:36:28.155 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56717.log: 81.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56717.log.gz 2024-09-12T16:36:28.155 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58427.log 2024-09-12T16:36:28.156 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.70249.log: 43.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70249.log.gz 2024-09-12T16:36:28.169 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58469.log 2024-09-12T16:36:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58427.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58427.log.gz 2024-09-12T16:36:28.184 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39943.log 2024-09-12T16:36:28.185 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58469.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58469.log.gz 2024-09-12T16:36:28.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63084.log 2024-09-12T16:36:28.194 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.39943.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39943.log.gz 2024-09-12T16:36:28.203 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38217.log 2024-09-12T16:36:28.204 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63084.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63084.log.gz 2024-09-12T16:36:28.219 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66525.log 2024-09-12T16:36:28.219 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38217.log: 74.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38217.log.gz 2024-09-12T16:36:28.228 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64226.log 2024-09-12T16:36:28.229 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66525.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66525.log.gz 2024-09-12T16:36:28.237 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45661.log 2024-09-12T16:36:28.238 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64226.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64226.log.gz 2024-09-12T16:36:28.249 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68543.log 2024-09-12T16:36:28.254 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45661.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45661.log.gz 2024-09-12T16:36:28.268 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70050.log 2024-09-12T16:36:28.268 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68543.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68543.log.gz 2024-09-12T16:36:28.285 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47849.log.gz 2024-09-12T16:36:28.285 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46790.log 2024-09-12T16:36:28.286 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70050.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70050.log.gz 2024-09-12T16:36:28.287 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53100.log 2024-09-12T16:36:28.288 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46790.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46790.log.gz 2024-09-12T16:36:28.288 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64480.log 2024-09-12T16:36:28.288 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53100.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53100.log.gz 2024-09-12T16:36:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51137.log 2024-09-12T16:36:28.290 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64480.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64480.log.gz 2024-09-12T16:36:28.291 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42067.log 2024-09-12T16:36:28.291 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51137.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51137.log.gz 2024-09-12T16:36:28.292 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40755.log 2024-09-12T16:36:28.294 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42067.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42067.log.gz 2024-09-12T16:36:28.295 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46219.log 2024-09-12T16:36:28.296 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40755.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40755.log.gz 2024-09-12T16:36:28.296 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47904.log 2024-09-12T16:36:28.297 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46219.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46219.log.gz 2024-09-12T16:36:28.298 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40060.log 2024-09-12T16:36:28.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47904.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47904.log.gz 2024-09-12T16:36:28.300 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56471.log 2024-09-12T16:36:28.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40060.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40060.log.gz 2024-09-12T16:36:28.302 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68880.log 2024-09-12T16:36:28.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56471.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56471.log.gz 2024-09-12T16:36:28.303 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62669.log 2024-09-12T16:36:28.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68880.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68880.log.gz 2024-09-12T16:36:28.305 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50379.log 2024-09-12T16:36:28.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62669.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62669.log.gz 2024-09-12T16:36:28.306 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69848.log 2024-09-12T16:36:28.321 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77199.log 2024-09-12T16:36:28.322 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69848.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69848.log.gz 2024-09-12T16:36:28.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62575.log 2024-09-12T16:36:28.336 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77199.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77199.log.gz 2024-09-12T16:36:28.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74199.log 2024-09-12T16:36:28.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62575.log: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60961.log 2024-09-12T16:36:28.343 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62575.log.gz 2024-09-12T16:36:28.345 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46449.log.gz 2024-09-12T16:36:28.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74199.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74199.log.gz 2024-09-12T16:36:28.348 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62717.log 2024-09-12T16:36:28.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60961.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60961.log.gz 2024-09-12T16:36:28.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81358.log 2024-09-12T16:36:28.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62717.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62717.log.gz 2024-09-12T16:36:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48504.log 2024-09-12T16:36:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81358.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81358.log.gz 2024-09-12T16:36:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67478.log 2024-09-12T16:36:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48504.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48504.log.gz 2024-09-12T16:36:28.352 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51367.log 2024-09-12T16:36:28.355 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67478.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67478.log.gz 2024-09-12T16:36:28.355 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55230.log 2024-09-12T16:36:28.356 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51367.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51367.log.gz 2024-09-12T16:36:28.356 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72232.log 2024-09-12T16:36:28.357 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55230.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55230.log.gz 2024-09-12T16:36:28.357 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65282.log 2024-09-12T16:36:28.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72232.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72232.log.gz 2024-09-12T16:36:28.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65158.log 2024-09-12T16:36:28.361 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65282.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65282.log.gz 2024-09-12T16:36:28.362 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67726.log 2024-09-12T16:36:28.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65158.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65158.log.gz 2024-09-12T16:36:28.365 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77493.log 2024-09-12T16:36:28.367 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67726.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67726.log.gz 2024-09-12T16:36:28.368 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59951.log 2024-09-12T16:36:28.368 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77493.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77493.log.gz 2024-09-12T16:36:28.369 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56512.log 2024-09-12T16:36:28.370 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59951.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59951.log.gz 2024-09-12T16:36:28.371 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54567.log 2024-09-12T16:36:28.374 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56512.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56512.log.gz 2024-09-12T16:36:28.374 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80014.log 2024-09-12T16:36:28.375 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54567.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54567.log.gz 2024-09-12T16:36:28.376 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52875.log 2024-09-12T16:36:28.376 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80014.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80014.log.gz 2024-09-12T16:36:28.377 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58587.log 2024-09-12T16:36:28.377 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52875.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52875.log.gz 2024-09-12T16:36:28.378 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44132.log 2024-09-12T16:36:28.379 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58587.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58587.log.gz 2024-09-12T16:36:28.380 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81526.log 2024-09-12T16:36:28.380 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44132.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44132.log.gz 2024-09-12T16:36:28.381 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56218.log 2024-09-12T16:36:28.381 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81526.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81526.log.gz 2024-09-12T16:36:28.382 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71758.log 2024-09-12T16:36:28.383 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56218.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56218.log.gz 2024-09-12T16:36:28.384 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45919.log 2024-09-12T16:36:28.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71758.log: 82.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71758.log.gz 2024-09-12T16:36:28.385 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45145.log 2024-09-12T16:36:28.386 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45919.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45919.log.gz 2024-09-12T16:36:28.387 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56399.log 2024-09-12T16:36:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45145.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45145.log.gz 2024-09-12T16:36:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71871.log 2024-09-12T16:36:28.395 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56399.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56399.log.gz 2024-09-12T16:36:28.395 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50300.log 2024-09-12T16:36:28.396 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71871.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71871.log.gz 2024-09-12T16:36:28.396 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76359.log 2024-09-12T16:36:28.411 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64785.log 2024-09-12T16:36:28.411 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76359.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76359.log.gz 2024-09-12T16:36:28.417 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72755.log.gz 2024-09-12T16:36:28.428 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50643.log 2024-09-12T16:36:28.428 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63340.log 2024-09-12T16:36:28.429 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69488.log 2024-09-12T16:36:28.429 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46832.log 2024-09-12T16:36:28.429 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.64785.log: -- /var/log/ceph/cluster2-client.admin.77745.log 2024-09-12T16:36:28.430 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63340.log: /var/log/ceph/cluster2-client.mirror.46832.log: /var/log/ceph/cluster2-client.mirror.50643.log: /var/log/ceph/cluster1-client.mirror.69488.log: 86.8% 89.7% 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64785.log.gz 2024-09-12T16:36:28.433 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63340.log.gz 2024-09-12T16:36:28.437 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69488.log.gz 2024-09-12T16:36:28.437 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44308.log.gz 2024-09-12T16:36:28.437 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38752.log 2024-09-12T16:36:28.438 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77745.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77745.log.gz 2024-09-12T16:36:28.438 INFO:teuthology.orchestra.run.smithi019.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46832.log.gz 2024-09-12T16:36:28.442 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48993.log.gz 2024-09-12T16:36:28.444 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72269.log 2024-09-12T16:36:28.444 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52674.log 2024-09-12T16:36:28.444 INFO:teuthology.orchestra.run.smithi019.stderr: 68.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38752.log.gz 2024-09-12T16:36:28.444 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57722.log 2024-09-12T16:36:28.448 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72269.log.gz 2024-09-12T16:36:28.448 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52674.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52674.log.gz 2024-09-12T16:36:28.448 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72854.log 2024-09-12T16:36:28.448 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69555.log 2024-09-12T16:36:28.448 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57722.log.gz 2024-09-12T16:36:28.448 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73927.log 2024-09-12T16:36:28.449 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69555.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72854.log.gz 2024-09-12T16:36:28.449 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69555.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55025.log 2024-09-12T16:36:28.449 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:28.449 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59592.log 2024-09-12T16:36:28.449 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73927.log: /var/log/ceph/cluster1-client.mirror.55025.log: 87.7% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73927.log.gz 2024-09-12T16:36:28.449 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55025.log.gz 2024-09-12T16:36:28.449 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78586.log 2024-09-12T16:36:28.452 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.59592.log: /var/log/ceph/cluster2-client.mirror.67561.log 2024-09-12T16:36:28.452 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78586.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78586.log.gz 2024-09-12T16:36:28.452 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45939.log 2024-09-12T16:36:28.452 INFO:teuthology.orchestra.run.smithi019.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59592.log.gz 2024-09-12T16:36:28.452 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67561.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67561.log.gz 2024-09-12T16:36:28.452 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80350.log 2024-09-12T16:36:28.452 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46239.log 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80350.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80350.log.gz 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45939.log.gz 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79804.log 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.46239.log: /var/log/ceph/cluster1-client.mirror.74155.log 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79804.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79804.log.gz 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46239.log.gz 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61234.log 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66835.log 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74155.log.gz 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72191.log 2024-09-12T16:36:28.459 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66835.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61234.log.gz 2024-09-12T16:36:28.460 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76023.log 2024-09-12T16:36:28.460 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66835.log.gz 2024-09-12T16:36:28.460 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43860.log 2024-09-12T16:36:28.460 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76023.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76023.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72191.log.gz 2024-09-12T16:36:28.460 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:28.460 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66856.log 2024-09-12T16:36:28.460 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.43860.log: /var/log/ceph/cluster2-client.mirror.59250.log 2024-09-12T16:36:28.462 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66856.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66856.log.gz 2024-09-12T16:36:28.462 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48721.log 2024-09-12T16:36:28.462 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59250.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59250.log.gz 2024-09-12T16:36:28.462 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45013.log 2024-09-12T16:36:28.464 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48721.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43860.log.gz 2024-09-12T16:36:28.472 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57628.log 2024-09-12T16:36:28.485 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56492.log 2024-09-12T16:36:28.485 INFO:teuthology.orchestra.run.smithi019.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45013.log.gz 2024-09-12T16:36:28.486 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79048.log 2024-09-12T16:36:28.486 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56492.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57628.log.gz 2024-09-12T16:36:28.488 INFO:teuthology.orchestra.run.smithi019.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56492.log.gz 2024-09-12T16:36:28.488 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50221.log 2024-09-12T16:36:28.489 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79048.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79048.log.gz 2024-09-12T16:36:28.489 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66153.log 2024-09-12T16:36:28.500 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64110.log 2024-09-12T16:36:28.501 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66153.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66153.log.gz 2024-09-12T16:36:28.513 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65760.log 2024-09-12T16:36:28.515 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64110.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64110.log.gz 2024-09-12T16:36:28.517 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46891.log.gz 2024-09-12T16:36:28.518 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63408.log 2024-09-12T16:36:28.520 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65760.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65760.log.gz 2024-09-12T16:36:28.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46997.log 2024-09-12T16:36:28.521 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63408.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63408.log.gz 2024-09-12T16:36:28.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68790.log 2024-09-12T16:36:28.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46997.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47478.log.gz 2024-09-12T16:36:28.529 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47790.log 2024-09-12T16:36:28.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68790.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68790.log.gz 2024-09-12T16:36:28.532 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79216.log 2024-09-12T16:36:28.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47790.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47790.log.gz 2024-09-12T16:36:28.532 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40104.log 2024-09-12T16:36:28.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79216.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79216.log.gz 2024-09-12T16:36:28.532 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45899.log 2024-09-12T16:36:28.532 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40104.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40104.log.gz 2024-09-12T16:36:28.533 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61138.log 2024-09-12T16:36:28.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45899.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45899.log.gz 2024-09-12T16:36:28.534 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46259.log 2024-09-12T16:36:28.535 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61138.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61138.log.gz 2024-09-12T16:36:28.536 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77283.log 2024-09-12T16:36:28.537 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46259.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46259.log.gz 2024-09-12T16:36:28.537 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64045.log 2024-09-12T16:36:28.538 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77283.log: 25.7% -- replaced with /var/log/ceph/cluster2-client.admin.77283.log.gz 2024-09-12T16:36:28.539 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62283.log 2024-09-12T16:36:28.540 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64045.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64045.log.gz 2024-09-12T16:36:28.540 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64827.log 2024-09-12T16:36:28.541 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62283.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62283.log.gz 2024-09-12T16:36:28.541 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43594.log 2024-09-12T16:36:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64827.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64827.log.gz 2024-09-12T16:36:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50337.log 2024-09-12T16:36:28.543 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43594.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43594.log.gz 2024-09-12T16:36:28.544 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43327.log 2024-09-12T16:36:28.546 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50337.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50337.log.gz 2024-09-12T16:36:28.546 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75603.log 2024-09-12T16:36:28.552 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43327.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43327.log.gz 2024-09-12T16:36:28.553 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42985.log 2024-09-12T16:36:28.553 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75603.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75603.log.gz 2024-09-12T16:36:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56534.log 2024-09-12T16:36:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42985.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42985.log.gz 2024-09-12T16:36:28.556 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69645.log 2024-09-12T16:36:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56534.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56534.log.gz 2024-09-12T16:36:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44420.log 2024-09-12T16:36:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69645.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69645.log.gz 2024-09-12T16:36:28.559 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68723.log 2024-09-12T16:36:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44420.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44420.log.gz 2024-09-12T16:36:28.560 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56170.log 2024-09-12T16:36:28.561 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68723.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68723.log.gz 2024-09-12T16:36:28.562 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70267.log 2024-09-12T16:36:28.562 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56170.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56170.log.gz 2024-09-12T16:36:28.563 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51533.log 2024-09-12T16:36:28.563 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.70267.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70267.log.gz 2024-09-12T16:36:28.565 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67892.log 2024-09-12T16:36:28.565 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51533.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51533.log.gz 2024-09-12T16:36:28.565 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40577.log 2024-09-12T16:36:28.567 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67892.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67892.log.gz 2024-09-12T16:36:28.567 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61117.log 2024-09-12T16:36:28.567 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40577.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40577.log.gz 2024-09-12T16:36:28.568 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73823.log 2024-09-12T16:36:28.570 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61117.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61117.log.gz 2024-09-12T16:36:28.570 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68273.log 2024-09-12T16:36:28.571 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73823.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73823.log.gz 2024-09-12T16:36:28.571 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62304.log 2024-09-12T16:36:28.573 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68273.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68273.log.gz 2024-09-12T16:36:28.573 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44671.log 2024-09-12T16:36:28.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62304.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62304.log.gz 2024-09-12T16:36:28.575 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56963.log 2024-09-12T16:36:28.576 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44671.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44671.log.gz 2024-09-12T16:36:28.577 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.49109.log 2024-09-12T16:36:28.578 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56963.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56963.log.gz 2024-09-12T16:36:28.578 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42814.log 2024-09-12T16:36:28.586 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.49109.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.49109.log.gz 2024-09-12T16:36:28.587 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47983.log 2024-09-12T16:36:28.594 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42814.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42814.log.gz 2024-09-12T16:36:28.594 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64202.log 2024-09-12T16:36:28.595 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47983.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50643.log.gz 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47983.log.gz 2024-09-12T16:36:28.595 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:28.595 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50880.log 2024-09-12T16:36:28.599 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.64202.log: /var/log/ceph/cluster2-client.admin.78123.log 2024-09-12T16:36:28.599 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64202.log.gz 2024-09-12T16:36:28.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68520.log 2024-09-12T16:36:28.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78123.log.gz 2024-09-12T16:36:28.600 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82450.log 2024-09-12T16:36:28.600 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68520.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68520.log.gz 2024-09-12T16:36:28.600 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57336.log 2024-09-12T16:36:28.600 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82450.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82450.log.gz 2024-09-12T16:36:28.600 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60573.log 2024-09-12T16:36:28.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57336.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57336.log.gz 2024-09-12T16:36:28.602 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61718.log 2024-09-12T16:36:28.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60573.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60573.log.gz 2024-09-12T16:36:28.602 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80224.log 2024-09-12T16:36:28.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61718.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61718.log.gz 2024-09-12T16:36:28.604 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52807.log 2024-09-12T16:36:28.604 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80224.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80224.log.gz 2024-09-12T16:36:28.604 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46119.log 2024-09-12T16:36:28.605 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52807.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52807.log.gz 2024-09-12T16:36:28.606 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68363.log 2024-09-12T16:36:28.606 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46119.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46119.log.gz 2024-09-12T16:36:28.607 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61938.log 2024-09-12T16:36:28.608 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68363.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68363.log.gz 2024-09-12T16:36:28.609 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50505.log 2024-09-12T16:36:28.609 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50379.log.gz 2024-09-12T16:36:28.609 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61938.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61938.log.gz 2024-09-12T16:36:28.611 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43972.log 2024-09-12T16:36:28.611 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64765.log 2024-09-12T16:36:28.611 INFO:teuthology.orchestra.run.smithi019.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50505.log.gz 2024-09-12T16:36:28.611 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.43972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51622.log 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.64765.log: -- replaced with /var/log/ceph/cluster1-client.admin.43972.log.gz 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82156.log 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64765.log.gz 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42372.log 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51622.log.gz 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63384.log 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42372.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82156.log.gz 2024-09-12T16:36:28.615 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67850.log 2024-09-12T16:36:28.616 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63384.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42372.log.gz 2024-09-12T16:36:28.616 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63384.log.gz 2024-09-12T16:36:28.616 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73882.log 2024-09-12T16:36:28.616 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74618.log 2024-09-12T16:36:28.617 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73882.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67850.log.gz 2024-09-12T16:36:28.618 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68115.log 2024-09-12T16:36:28.618 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74618.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74618.log.gz 2024-09-12T16:36:28.619 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50564.log 2024-09-12T16:36:28.620 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68115.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68115.log.gz 2024-09-12T16:36:28.620 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69758.log 2024-09-12T16:36:28.636 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74641.log 2024-09-12T16:36:28.637 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69758.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69758.log.gz 2024-09-12T16:36:28.650 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39809.log 2024-09-12T16:36:28.650 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74641.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74641.log.gz 2024-09-12T16:36:28.661 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43167.log 2024-09-12T16:36:28.662 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.39809.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39809.log.gz 2024-09-12T16:36:28.675 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65386.log 2024-09-12T16:36:28.676 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43167.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43167.log.gz 2024-09-12T16:36:28.684 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36189.log 2024-09-12T16:36:28.693 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50179.log 2024-09-12T16:36:28.697 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.36189.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65386.log.gz 2024-09-12T16:36:28.702 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53237.log 2024-09-12T16:36:28.706 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50179.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36189.log.gz 2024-09-12T16:36:28.716 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50179.log.gz 2024-09-12T16:36:28.719 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58218.log 2024-09-12T16:36:28.721 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53237.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53237.log.gz 2024-09-12T16:36:28.732 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50300.log.gz 2024-09-12T16:36:28.733 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72500.log 2024-09-12T16:36:28.737 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58218.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58218.log.gz 2024-09-12T16:36:28.738 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65698.log 2024-09-12T16:36:28.738 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72500.log: 86.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72500.log.gz 2024-09-12T16:36:28.738 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43550.log 2024-09-12T16:36:28.741 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65698.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65698.log.gz 2024-09-12T16:36:28.741 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.51533.log 2024-09-12T16:36:28.743 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43550.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43550.log.gz 2024-09-12T16:36:28.743 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37456.log 2024-09-12T16:36:28.748 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.1.51533.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.51533.log.gz 2024-09-12T16:36:28.748 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56943.log 2024-09-12T16:36:28.748 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37456.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37456.log.gz 2024-09-12T16:36:28.749 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59681.log 2024-09-12T16:36:28.750 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56943.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56943.log.gz 2024-09-12T16:36:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80476.log 2024-09-12T16:36:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59681.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59681.log.gz 2024-09-12T16:36:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54588.log 2024-09-12T16:36:28.753 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80476.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80476.log.gz 2024-09-12T16:36:28.753 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38135.log 2024-09-12T16:36:28.756 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54588.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54588.log.gz 2024-09-12T16:36:28.756 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.45013.log 2024-09-12T16:36:28.757 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.38135.log: 72.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38135.log.gz 2024-09-12T16:36:28.758 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74727.log 2024-09-12T16:36:28.767 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.2.45013.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50880.log.gz 2024-09-12T16:36:28.767 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72696.log 2024-09-12T16:36:28.779 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46997.log.gz 2024-09-12T16:36:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53349.log 2024-09-12T16:36:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47614.log 2024-09-12T16:36:28.779 INFO:teuthology.orchestra.run.smithi019.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74727.log.gz 2024-09-12T16:36:28.779 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53349.log: /var/log/ceph/cluster2-client.mirror.72696.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72696.log.gz 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53349.log.gz 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61644.log 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54519.log 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61644.log: /var/log/ceph/cluster2-client.mirror.47614.log: 87.4% 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47614.log.gz 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61644.log.gz 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75687.log 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77115.log 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54519.log.gz 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73969.log 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77115.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75687.log.gz 2024-09-12T16:36:28.780 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77115.log.gz 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40285.log 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63973.log 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73969.log.gz 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47673.log 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63973.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63973.log.gz 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75939.log 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47673.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47673.log.gz 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48452.log 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75939.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75939.log.gz 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68453.log 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48452.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48452.log.gz 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41019.log 2024-09-12T16:36:28.781 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68453.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68453.log.gz 2024-09-12T16:36:28.783 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44735.log 2024-09-12T16:36:28.783 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41019.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41019.log.gz 2024-09-12T16:36:28.783 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41067.log 2024-09-12T16:36:28.788 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44735.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44735.log.gz 2024-09-12T16:36:28.788 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62054.log 2024-09-12T16:36:28.789 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41067.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41067.log.gz 2024-09-12T16:36:28.789 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74497.log 2024-09-12T16:36:28.791 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62054.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62054.log.gz 2024-09-12T16:36:28.791 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56146.log 2024-09-12T16:36:28.791 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74497.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74497.log.gz 2024-09-12T16:36:28.792 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42633.log 2024-09-12T16:36:28.793 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56146.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56146.log.gz 2024-09-12T16:36:28.794 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66959.log 2024-09-12T16:36:28.796 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42633.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42633.log.gz 2024-09-12T16:36:28.796 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44911.log 2024-09-12T16:36:28.797 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66959.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66959.log.gz 2024-09-12T16:36:28.797 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40361.log 2024-09-12T16:36:28.798 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44911.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44911.log.gz 2024-09-12T16:36:28.799 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66215.log 2024-09-12T16:36:28.801 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40361.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40361.log.gz 2024-09-12T16:36:28.801 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72149.log 2024-09-12T16:36:28.802 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66215.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66215.log.gz 2024-09-12T16:36:28.804 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62991.log 2024-09-12T16:36:28.804 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72149.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72149.log.gz 2024-09-12T16:36:28.805 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50162.log 2024-09-12T16:36:28.807 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62991.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62991.log.gz 2024-09-12T16:36:28.807 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72654.log 2024-09-12T16:36:28.809 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50162.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50162.log.gz 2024-09-12T16:36:28.809 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44551.log 2024-09-12T16:36:28.810 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72654.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72654.log.gz 2024-09-12T16:36:28.810 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65241.log 2024-09-12T16:36:28.811 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44551.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44551.log.gz 2024-09-12T16:36:28.812 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44357.log 2024-09-12T16:36:28.813 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65241.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65241.log.gz 2024-09-12T16:36:28.813 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82534.log 2024-09-12T16:36:28.814 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.44357.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44357.log.gz 2024-09-12T16:36:28.815 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59702.log 2024-09-12T16:36:28.815 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82534.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82534.log.gz 2024-09-12T16:36:28.816 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42943.log 2024-09-12T16:36:28.818 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59702.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59702.log.gz 2024-09-12T16:36:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77997.log 2024-09-12T16:36:28.825 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42943.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42943.log.gz 2024-09-12T16:36:28.826 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47715.log 2024-09-12T16:36:28.826 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77997.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77997.log.gz 2024-09-12T16:36:28.827 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63012.log 2024-09-12T16:36:28.840 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45446.log 2024-09-12T16:36:28.842 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63012.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63012.log.gz 2024-09-12T16:36:28.845 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73882.log.gz 2024-09-12T16:36:28.846 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59367.log 2024-09-12T16:36:28.852 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45446.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45446.log.gz 2024-09-12T16:36:28.852 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48003.log 2024-09-12T16:36:28.855 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59367.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59367.log.gz 2024-09-12T16:36:28.855 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74538.log 2024-09-12T16:36:28.855 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48003.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48003.log.gz 2024-09-12T16:36:28.855 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46702.log 2024-09-12T16:36:28.856 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74538.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74538.log.gz 2024-09-12T16:36:28.857 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49109.log 2024-09-12T16:36:28.870 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71364.log 2024-09-12T16:36:28.871 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49109.log: 86.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49109.log.gz 2024-09-12T16:36:28.877 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52899.log 2024-09-12T16:36:28.890 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.42851.log 2024-09-12T16:36:28.891 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52899.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52899.log.gz 2024-09-12T16:36:28.891 INFO:teuthology.orchestra.run.smithi019.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71364.log.gz 2024-09-12T16:36:28.904 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42394.log 2024-09-12T16:36:28.904 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50221.log.gz 2024-09-12T16:36:28.905 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.42851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62211.log 2024-09-12T16:36:28.908 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42394.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.42851.log.gz 2024-09-12T16:36:28.917 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40176.log 2024-09-12T16:36:28.917 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62211.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62211.log.gz 2024-09-12T16:36:28.930 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55299.log 2024-09-12T16:36:28.931 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40176.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40176.log.gz 2024-09-12T16:36:28.932 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39047.log 2024-09-12T16:36:28.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55299.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55299.log.gz 2024-09-12T16:36:28.940 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48721.log.gz 2024-09-12T16:36:28.940 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42432.log 2024-09-12T16:36:28.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.39047.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39047.log.gz 2024-09-12T16:36:28.942 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-12T16:36:28.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.42432.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42432.log.gz 2024-09-12T16:36:28.944 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52968.log 2024-09-12T16:36:28.954 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64436.log 2024-09-12T16:36:28.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52968.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51835.log 2024-09-12T16:36:28.961 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52968.log.gz 2024-09-12T16:36:28.974 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58611.log 2024-09-12T16:36:28.974 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51835.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64436.log.gz 2024-09-12T16:36:28.975 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51835.log.gz 2024-09-12T16:36:28.986 INFO:teuthology.orchestra.run.smithi019.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.45013.log.gz 2024-09-12T16:36:28.986 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49211.log 2024-09-12T16:36:28.986 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58611.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58611.log.gz 2024-09-12T16:36:28.988 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.74236.log 2024-09-12T16:36:28.988 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49211.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49211.log.gz 2024-09-12T16:36:28.989 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74013.log 2024-09-12T16:36:28.991 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.2.74236.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.74236.log.gz 2024-09-12T16:36:28.992 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74178.log 2024-09-12T16:36:28.992 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74013.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74013.log.gz 2024-09-12T16:36:28.993 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78628.log 2024-09-12T16:36:28.995 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74178.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74178.log.gz 2024-09-12T16:36:28.995 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69668.log 2024-09-12T16:36:28.995 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78628.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78628.log.gz 2024-09-12T16:36:28.996 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38609.log 2024-09-12T16:36:28.996 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69668.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69668.log.gz 2024-09-12T16:36:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62464.log 2024-09-12T16:36:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.38609.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38609.log.gz 2024-09-12T16:36:28.998 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39899.log 2024-09-12T16:36:28.999 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50564.log.gz 2024-09-12T16:36:28.999 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62464.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62464.log.gz 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49030.log 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.39899.log: -- /var/log/ceph/cluster1-client.mirror.70005.log 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39899.log.gz 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57059.log 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70005.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70005.log.gz 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49030.log.gz 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78039.log 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74821.log 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57059.log.gz 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78039.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78039.log.gz 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64132.log 2024-09-12T16:36:29.004 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56281.log 2024-09-12T16:36:29.005 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74821.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74821.log.gz 2024-09-12T16:36:29.005 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68813.log 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56281.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64132.log.gz 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56281.log.gz 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60621.log 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81610.log 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68813.log: /var/log/ceph/cluster1-client.mirror.60621.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68813.log.gz 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80938.log 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60621.log.gz 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81610.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81610.log.gz 2024-09-12T16:36:29.007 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55712.log 2024-09-12T16:36:29.008 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52245.log 2024-09-12T16:36:29.009 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80938.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80938.log.gz 2024-09-12T16:36:29.009 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42227.log 2024-09-12T16:36:29.009 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55712.log.gz 2024-09-12T16:36:29.010 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52245.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52245.log.gz 2024-09-12T16:36:29.010 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47141.log 2024-09-12T16:36:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41245.log 2024-09-12T16:36:29.011 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42227.log: /var/log/ceph/cluster2-client.mirror.47141.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42227.log.gz 2024-09-12T16:36:29.012 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58700.log 2024-09-12T16:36:29.012 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41245.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47141.log.gz 2024-09-12T16:36:29.012 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41245.log.gz 2024-09-12T16:36:29.013 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54977.log 2024-09-12T16:36:29.013 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51409.log 2024-09-12T16:36:29.014 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58700.log: /var/log/ceph/cluster1-client.mirror.54977.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54977.log.gz 2024-09-12T16:36:29.015 INFO:teuthology.orchestra.run.smithi019.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58700.log.gz 2024-09-12T16:36:29.016 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40335.log 2024-09-12T16:36:29.016 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45532.log 2024-09-12T16:36:29.017 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51409.log: /var/log/ceph/cluster1-client.admin.40335.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.40335.log.gz 2024-09-12T16:36:29.033 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67416.log 2024-09-12T16:36:29.039 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45532.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45532.log.gz 2024-09-12T16:36:29.046 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69690.log 2024-09-12T16:36:29.051 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67416.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67416.log.gz 2024-09-12T16:36:29.051 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50821.log 2024-09-12T16:36:29.051 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69690.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69690.log.gz 2024-09-12T16:36:29.051 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47456.log 2024-09-12T16:36:29.051 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50821.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50821.log.gz 2024-09-12T16:36:29.053 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73367.log 2024-09-12T16:36:29.053 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47456.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47456.log.gz 2024-09-12T16:36:29.053 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41629.log 2024-09-12T16:36:29.054 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73367.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73367.log.gz 2024-09-12T16:36:29.054 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41735.log 2024-09-12T16:36:29.054 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41629.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41629.log.gz 2024-09-12T16:36:29.055 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57377.log 2024-09-12T16:36:29.057 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41735.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41735.log.gz 2024-09-12T16:36:29.057 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62095.log 2024-09-12T16:36:29.059 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57377.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57377.log.gz 2024-09-12T16:36:29.059 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41160.log 2024-09-12T16:36:29.062 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62095.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62095.log.gz 2024-09-12T16:36:29.062 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.38313.log 2024-09-12T16:36:29.068 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41160.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41160.log.gz 2024-09-12T16:36:29.068 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63128.log 2024-09-12T16:36:29.074 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.2.38313.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.38313.log.gz 2024-09-12T16:36:29.074 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52104.log 2024-09-12T16:36:29.075 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63128.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63128.log.gz 2024-09-12T16:36:29.076 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71528.log 2024-09-12T16:36:29.076 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52104.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52104.log.gz 2024-09-12T16:36:29.077 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64364.log 2024-09-12T16:36:29.078 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71528.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71528.log.gz 2024-09-12T16:36:29.078 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37288.log 2024-09-12T16:36:29.081 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64364.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64364.log.gz 2024-09-12T16:36:29.081 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74575.log 2024-09-12T16:36:29.082 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37288.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37288.log.gz 2024-09-12T16:36:29.083 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66752.log 2024-09-12T16:36:29.083 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74575.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74575.log.gz 2024-09-12T16:36:29.084 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.53695.log 2024-09-12T16:36:29.087 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66752.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66752.log.gz 2024-09-12T16:36:29.087 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62030.log 2024-09-12T16:36:29.092 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.2.53695.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.53695.log.gz 2024-09-12T16:36:29.092 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82240.log 2024-09-12T16:36:29.093 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62030.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62030.log.gz 2024-09-12T16:36:29.094 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68475.log 2024-09-12T16:36:29.095 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82240.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82240.log.gz 2024-09-12T16:36:29.095 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80266.log 2024-09-12T16:36:29.096 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68475.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68475.log.gz 2024-09-12T16:36:29.096 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38585.log 2024-09-12T16:36:29.097 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80266.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80266.log.gz 2024-09-12T16:36:29.099 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73904.log 2024-09-12T16:36:29.099 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.38585.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38585.log.gz 2024-09-12T16:36:29.099 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59117.log 2024-09-12T16:36:29.099 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73904.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73904.log.gz 2024-09-12T16:36:29.100 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54355.log 2024-09-12T16:36:29.101 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59117.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59117.log.gz 2024-09-12T16:36:29.101 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37666.log 2024-09-12T16:36:29.104 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54355.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54355.log.gz 2024-09-12T16:36:29.104 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44819.log 2024-09-12T16:36:29.105 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.37666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37666.log.gz 2024-09-12T16:36:29.105 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43304.log 2024-09-12T16:36:29.107 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44819.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44819.log.gz 2024-09-12T16:36:29.107 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46342.log 2024-09-12T16:36:29.113 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43304.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43304.log.gz 2024-09-12T16:36:29.114 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40080.log 2024-09-12T16:36:29.115 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.46342.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46342.log.gz 2024-09-12T16:36:29.115 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52698.log 2024-09-12T16:36:29.116 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40080.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40080.log.gz 2024-09-12T16:36:29.116 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64910.log 2024-09-12T16:36:29.117 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52698.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52698.log.gz 2024-09-12T16:36:29.118 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67768.log 2024-09-12T16:36:29.121 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64910.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64910.log.gz 2024-09-12T16:36:29.121 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63456.log 2024-09-12T16:36:29.122 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67768.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67768.log.gz 2024-09-12T16:36:29.123 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.42522.log 2024-09-12T16:36:29.123 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63456.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63456.log.gz 2024-09-12T16:36:29.124 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53506.log 2024-09-12T16:36:29.127 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.2.42522.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.42522.log.gz 2024-09-12T16:36:29.128 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62533.log 2024-09-12T16:36:29.129 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53506.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53506.log.gz 2024-09-12T16:36:29.129 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56259.log 2024-09-12T16:36:29.131 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62533.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62533.log.gz 2024-09-12T16:36:29.132 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40733.log 2024-09-12T16:36:29.135 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56259.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56259.log.gz 2024-09-12T16:36:29.135 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62741.log 2024-09-12T16:36:29.141 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40733.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40733.log.gz 2024-09-12T16:36:29.142 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37582.log 2024-09-12T16:36:29.143 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62741.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62741.log.gz 2024-09-12T16:36:29.143 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71429.log 2024-09-12T16:36:29.144 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37582.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37582.log.gz 2024-09-12T16:36:29.144 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76149.log 2024-09-12T16:36:29.146 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71429.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71429.log.gz 2024-09-12T16:36:29.146 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65344.log 2024-09-12T16:36:29.146 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76149.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76149.log.gz 2024-09-12T16:36:29.147 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77451.log 2024-09-12T16:36:29.150 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65344.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65344.log.gz 2024-09-12T16:36:29.150 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75897.log 2024-09-12T16:36:29.151 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77451.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77451.log.gz 2024-09-12T16:36:29.152 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48563.log 2024-09-12T16:36:29.152 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75897.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75897.log.gz 2024-09-12T16:36:29.152 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67062.log 2024-09-12T16:36:29.168 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48563.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47203.log 2024-09-12T16:36:29.171 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67062.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67062.log.gz 2024-09-12T16:36:29.184 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56028.log 2024-09-12T16:36:29.186 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47203.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47203.log.gz 2024-09-12T16:36:29.186 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74796.log 2024-09-12T16:36:29.186 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51409.log.gz 2024-09-12T16:36:29.187 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56028.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56028.log.gz 2024-09-12T16:36:29.189 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40285.log.gz 2024-09-12T16:36:29.189 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46139.log 2024-09-12T16:36:29.189 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43570.log 2024-09-12T16:36:29.190 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74796.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74796.log.gz 2024-09-12T16:36:29.190 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67251.log 2024-09-12T16:36:29.190 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43570.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43570.log.gz 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46139.log.gz 2024-09-12T16:36:29.190 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:29.190 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61528.log 2024-09-12T16:36:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.67251.log: -- /var/log/ceph/cluster2-client.mirror.60272.log 2024-09-12T16:36:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61528.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67251.log.gz 2024-09-12T16:36:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62762.log 2024-09-12T16:36:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60272.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60272.log.gz 2024-09-12T16:36:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50663.log 2024-09-12T16:36:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62762.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61528.log.gz 2024-09-12T16:36:29.194 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48813.log 2024-09-12T16:36:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50663.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50663.log.gz 2024-09-12T16:36:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44933.log 2024-09-12T16:36:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48813.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62762.log.gz 2024-09-12T16:36:29.196 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48813.log.gz 2024-09-12T16:36:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38829.log 2024-09-12T16:36:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.44933.log: /var/log/ceph/cluster2-client.mirror.66236.log 2024-09-12T16:36:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.38829.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38829.log.gz 2024-09-12T16:36:29.198 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48679.log 2024-09-12T16:36:29.198 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66236.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66236.log.gz 2024-09-12T16:36:29.198 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50083.log 2024-09-12T16:36:29.200 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48679.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48679.log.gz 2024-09-12T16:36:29.200 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59069.log 2024-09-12T16:36:29.202 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50083.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50083.log.gz 2024-09-12T16:36:29.202 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44933.log.gz 2024-09-12T16:36:29.202 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49072.log 2024-09-12T16:36:29.202 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47966.log 2024-09-12T16:36:29.202 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59069.log: /var/log/ceph/cluster2-client.mirror.49072.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59069.log.gz 2024-09-12T16:36:29.206 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52827.log 2024-09-12T16:36:29.206 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47966.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47966.log.gz 2024-09-12T16:36:29.206 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32621.log 2024-09-12T16:36:29.206 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52827.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52827.log.gz 2024-09-12T16:36:29.206 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48699.log 2024-09-12T16:36:29.207 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.32621.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32621.log.gz 2024-09-12T16:36:29.207 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48247.log 2024-09-12T16:36:29.207 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48699.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48699.log.gz 2024-09-12T16:36:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53056.log 2024-09-12T16:36:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48247.log: 85.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48247.log.gz 2024-09-12T16:36:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48835.log 2024-09-12T16:36:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53056.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53056.log.gz 2024-09-12T16:36:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80686.log 2024-09-12T16:36:29.219 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45254.log 2024-09-12T16:36:29.220 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80686.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80686.log.gz 2024-09-12T16:36:29.235 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76611.log 2024-09-12T16:36:29.241 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45254.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45254.log.gz 2024-09-12T16:36:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69713.log 2024-09-12T16:36:29.250 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76611.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76611.log.gz 2024-09-12T16:36:29.250 INFO:teuthology.orchestra.run.smithi019.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47715.log.gz 2024-09-12T16:36:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81316.log 2024-09-12T16:36:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69713.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69713.log.gz 2024-09-12T16:36:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74683.log 2024-09-12T16:36:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81316.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81316.log.gz 2024-09-12T16:36:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38803.log 2024-09-12T16:36:29.253 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74683.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74683.log.gz 2024-09-12T16:36:29.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55664.log 2024-09-12T16:36:29.257 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.38803.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38803.log.gz 2024-09-12T16:36:29.257 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69128.log 2024-09-12T16:36:29.257 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55664.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55664.log.gz 2024-09-12T16:36:29.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65138.log 2024-09-12T16:36:29.258 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69128.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69128.log.gz 2024-09-12T16:36:29.258 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41139.log 2024-09-12T16:36:29.258 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65138.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65138.log.gz 2024-09-12T16:36:29.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57104.log 2024-09-12T16:36:29.260 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41139.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41139.log.gz 2024-09-12T16:36:29.260 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50278.log 2024-09-12T16:36:29.262 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57104.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57104.log.gz 2024-09-12T16:36:29.263 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46427.log 2024-09-12T16:36:29.264 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50278.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50278.log.gz 2024-09-12T16:36:29.264 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69263.log 2024-09-12T16:36:29.265 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46427.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46427.log.gz 2024-09-12T16:36:29.265 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65179.log 2024-09-12T16:36:29.266 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69263.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69263.log.gz 2024-09-12T16:36:29.266 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81946.log 2024-09-12T16:36:29.269 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65179.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65179.log.gz 2024-09-12T16:36:29.269 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58792.log 2024-09-12T16:36:29.269 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81946.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81946.log.gz 2024-09-12T16:36:29.270 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45999.log 2024-09-12T16:36:29.271 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58792.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58792.log.gz 2024-09-12T16:36:29.271 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64931.log 2024-09-12T16:36:29.272 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45999.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45999.log.gz 2024-09-12T16:36:29.273 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43659.log 2024-09-12T16:36:29.274 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64931.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64931.log.gz 2024-09-12T16:36:29.274 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82576.log 2024-09-12T16:36:29.275 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43659.log: 81.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43659.log.gz 2024-09-12T16:36:29.277 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66732.log 2024-09-12T16:36:29.277 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82576.log.gz 2024-09-12T16:36:29.277 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69893.log 2024-09-12T16:36:29.278 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66732.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66732.log.gz 2024-09-12T16:36:29.278 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68138.log 2024-09-12T16:36:29.279 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69893.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69893.log.gz 2024-09-12T16:36:29.280 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38152.log 2024-09-12T16:36:29.280 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68138.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68138.log.gz 2024-09-12T16:36:29.281 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49307.log 2024-09-12T16:36:29.290 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.1.38152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78333.log 2024-09-12T16:36:29.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49307.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46702.log.gz 2024-09-12T16:36:29.298 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.50416.log 2024-09-12T16:36:29.298 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78333.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78333.log.gz 2024-09-12T16:36:29.299 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49307.log.gz 2024-09-12T16:36:29.299 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55182.log 2024-09-12T16:36:29.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.1.50416.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.50416.log.gz 2024-09-12T16:36:29.307 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60549.log 2024-09-12T16:36:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55182.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55182.log.gz 2024-09-12T16:36:29.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60180.log 2024-09-12T16:36:29.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60549.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60549.log.gz 2024-09-12T16:36:29.309 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49327.log 2024-09-12T16:36:29.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60180.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60180.log.gz 2024-09-12T16:36:29.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37750.log 2024-09-12T16:36:29.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49327.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49327.log.gz 2024-09-12T16:36:29.314 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61463.log 2024-09-12T16:36:29.315 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.37750.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37750.log.gz 2024-09-12T16:36:29.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63860.log 2024-09-12T16:36:29.316 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61463.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61463.log.gz 2024-09-12T16:36:29.316 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67685.log 2024-09-12T16:36:29.318 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63860.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63860.log.gz 2024-09-12T16:36:29.319 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79090.log 2024-09-12T16:36:29.320 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67685.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67685.log.gz 2024-09-12T16:36:29.321 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-12T16:36:29.322 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79090.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79090.log.gz 2024-09-12T16:36:29.322 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48373.log 2024-09-12T16:36:29.335 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44531.log 2024-09-12T16:36:29.336 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48373.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48373.log.gz 2024-09-12T16:36:29.345 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42394.log.gz 2024-09-12T16:36:29.347 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49916.log 2024-09-12T16:36:29.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44531.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44531.log.gz 2024-09-12T16:36:29.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68993.log 2024-09-12T16:36:29.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49916.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49916.log.gz 2024-09-12T16:36:29.350 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65926.log 2024-09-12T16:36:29.351 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68993.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68993.log.gz 2024-09-12T16:36:29.351 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63224.log 2024-09-12T16:36:29.352 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65926.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65926.log.gz 2024-09-12T16:36:29.353 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43772.log 2024-09-12T16:36:29.353 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63224.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63224.log.gz 2024-09-12T16:36:29.354 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79384.log 2024-09-12T16:36:29.355 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43772.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43772.log.gz 2024-09-12T16:36:29.356 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71652.log 2024-09-12T16:36:29.356 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79384.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79384.log.gz 2024-09-12T16:36:29.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43902.log 2024-09-12T16:36:29.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71652.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71652.log.gz 2024-09-12T16:36:29.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50021.log 2024-09-12T16:36:29.360 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43902.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43902.log.gz 2024-09-12T16:36:29.360 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48563.log.gz 2024-09-12T16:36:29.360 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.42459.log 2024-09-12T16:36:29.360 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65220.log 2024-09-12T16:36:29.361 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50021.log: /var/log/ceph/cluster1-client.admin.42459.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42459.log.gz 2024-09-12T16:36:29.361 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50021.log.gz 2024-09-12T16:36:29.362 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78291.log 2024-09-12T16:36:29.364 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65220.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65220.log.gz 2024-09-12T16:36:29.364 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54748.log 2024-09-12T16:36:29.366 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48104.log 2024-09-12T16:36:29.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78291.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78291.log.gz 2024-09-12T16:36:29.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78249.log 2024-09-12T16:36:29.366 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54748.log.gz 2024-09-12T16:36:29.374 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41378.log 2024-09-12T16:36:29.375 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78249.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78249.log.gz 2024-09-12T16:36:29.375 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41693.log 2024-09-12T16:36:29.375 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41378.log: 81.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41378.log.gz 2024-09-12T16:36:29.376 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57083.log 2024-09-12T16:36:29.383 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.41693.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41693.log.gz 2024-09-12T16:36:29.383 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66628.log 2024-09-12T16:36:29.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57083.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57083.log.gz 2024-09-12T16:36:29.384 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44040.log 2024-09-12T16:36:29.387 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66628.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66628.log.gz 2024-09-12T16:36:29.388 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50142.log 2024-09-12T16:36:29.390 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44040.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44040.log.gz 2024-09-12T16:36:29.390 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51659.log 2024-09-12T16:36:29.409 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76695.log 2024-09-12T16:36:29.410 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51659.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51659.log.gz 2024-09-12T16:36:29.423 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43007.log 2024-09-12T16:36:29.424 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76695.log.gz 2024-09-12T16:36:29.430 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63104.log 2024-09-12T16:36:29.439 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52200.log 2024-09-12T16:36:29.440 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63104.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63104.log.gz 2024-09-12T16:36:29.454 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42851.log 2024-09-12T16:36:29.455 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52200.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52200.log.gz 2024-09-12T16:36:29.463 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42522.log 2024-09-12T16:36:29.463 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42851.log: 73.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42851.log.gz 2024-09-12T16:36:29.481 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44221.log 2024-09-12T16:36:29.481 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.42522.log: 81.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42522.log.gz 2024-09-12T16:36:29.488 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68408.log 2024-09-12T16:36:29.494 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44221.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44221.log.gz 2024-09-12T16:36:29.500 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59973.log 2024-09-12T16:36:29.501 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68408.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68408.log.gz 2024-09-12T16:36:29.516 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60023.log 2024-09-12T16:36:29.517 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59973.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59973.log.gz 2024-09-12T16:36:29.529 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54933.log 2024-09-12T16:36:29.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60023.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60023.log.gz 2024-09-12T16:36:29.549 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64250.log 2024-09-12T16:36:29.550 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54933.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54933.log.gz 2024-09-12T16:36:29.552 INFO:teuthology.orchestra.run.smithi019.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48835.log.gz 2024-09-12T16:36:29.561 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66360.log 2024-09-12T16:36:29.562 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64250.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64250.log.gz 2024-09-12T16:36:29.568 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50142.log.gz 2024-09-12T16:36:29.570 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44887.log 2024-09-12T16:36:29.571 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66360.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66360.log.gz 2024-09-12T16:36:29.572 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67458.log 2024-09-12T16:36:29.573 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44887.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44887.log.gz 2024-09-12T16:36:29.574 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46390.log 2024-09-12T16:36:29.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67458.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67458.log.gz 2024-09-12T16:36:29.575 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72061.log 2024-09-12T16:36:29.576 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46390.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46390.log.gz 2024-09-12T16:36:29.579 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76569.log 2024-09-12T16:36:29.579 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72061.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72061.log.gz 2024-09-12T16:36:29.579 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52176.log 2024-09-12T16:36:29.579 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76569.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76569.log.gz 2024-09-12T16:36:29.579 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80812.log 2024-09-12T16:36:29.583 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52176.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52176.log.gz 2024-09-12T16:36:29.583 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60874.log 2024-09-12T16:36:29.583 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80812.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80812.log.gz 2024-09-12T16:36:29.583 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47062.log 2024-09-12T16:36:29.583 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60874.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60874.log.gz 2024-09-12T16:36:29.583 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60597.log 2024-09-12T16:36:29.586 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47062.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47062.log.gz 2024-09-12T16:36:29.586 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65117.log 2024-09-12T16:36:29.586 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60597.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60597.log.gz 2024-09-12T16:36:29.586 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64344.log 2024-09-12T16:36:29.587 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65117.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65117.log.gz 2024-09-12T16:36:29.588 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82282.log 2024-09-12T16:36:29.589 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64344.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38152.log.gz 2024-09-12T16:36:29.589 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64344.log.gz 2024-09-12T16:36:29.591 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60252.log 2024-09-12T16:36:29.591 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82282.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82282.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46039.log 2024-09-12T16:36:29.591 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:29.591 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45747.log 2024-09-12T16:36:29.593 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46039.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60252.log.gz 2024-09-12T16:36:29.593 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46039.log.gz 2024-09-12T16:36:29.593 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53148.log 2024-09-12T16:36:29.593 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44839.log 2024-09-12T16:36:29.594 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53148.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53148.log.gz 2024-09-12T16:36:29.594 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56851.log 2024-09-12T16:36:29.594 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44839.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44839.log.gz 2024-09-12T16:36:29.595 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67230.log 2024-09-12T16:36:29.597 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56851.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56851.log.gz 2024-09-12T16:36:29.597 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53989.log 2024-09-12T16:36:29.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67230.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45747.log.gz 2024-09-12T16:36:29.599 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72312.log 2024-09-12T16:36:29.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53989.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53989.log.gz 2024-09-12T16:36:29.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58198.log 2024-09-12T16:36:29.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72312.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67230.log.gz 2024-09-12T16:36:29.601 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72312.log.gz 2024-09-12T16:36:29.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66050.log 2024-09-12T16:36:29.602 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.58198.log: /var/log/ceph/cluster2-client.mirror.66380.log 2024-09-12T16:36:29.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66050.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66050.log.gz 2024-09-12T16:36:29.604 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57449.log 2024-09-12T16:36:29.604 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58198.log.gz 2024-09-12T16:36:29.606 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66380.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66380.log.gz 2024-09-12T16:36:29.606 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45125.log 2024-09-12T16:36:29.607 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.57449.log: /var/log/ceph/cluster2-client.mirror.47262.log 2024-09-12T16:36:29.608 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45125.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57449.log.gz 2024-09-12T16:36:29.608 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55090.log 2024-09-12T16:36:29.613 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47262.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45125.log.gz 2024-09-12T16:36:29.613 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65905.log 2024-09-12T16:36:29.615 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55090.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55090.log.gz 2024-09-12T16:36:29.616 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45274.log 2024-09-12T16:36:29.617 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65905.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65905.log.gz 2024-09-12T16:36:29.617 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43924.log 2024-09-12T16:36:29.623 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45274.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45274.log.gz 2024-09-12T16:36:29.624 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-12T16:36:29.636 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61508.log 2024-09-12T16:36:29.649 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71991.log 2024-09-12T16:36:29.651 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61508.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61508.log.gz 2024-09-12T16:36:29.664 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43945.log 2024-09-12T16:36:29.665 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71991.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71991.log.gz 2024-09-12T16:36:29.671 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63432.log 2024-09-12T16:36:29.671 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.43945.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.43945.log.gz 2024-09-12T16:36:29.681 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53460.log 2024-09-12T16:36:29.681 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63432.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63432.log.gz 2024-09-12T16:36:29.684 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49072.log.gz 2024-09-12T16:36:29.690 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60202.log 2024-09-12T16:36:29.704 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53460.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53460.log.gz 2024-09-12T16:36:29.710 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79468.log 2024-09-12T16:36:29.711 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60202.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60202.log.gz 2024-09-12T16:36:29.725 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60320.log 2024-09-12T16:36:29.725 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79468.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79468.log.gz 2024-09-12T16:36:29.738 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69015.log 2024-09-12T16:36:29.739 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60320.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60320.log.gz 2024-09-12T16:36:29.760 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48353.log 2024-09-12T16:36:29.760 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69015.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69015.log.gz 2024-09-12T16:36:29.768 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74597.log 2024-09-12T16:36:29.769 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48353.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48353.log.gz 2024-09-12T16:36:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52016.log 2024-09-12T16:36:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58539.log 2024-09-12T16:36:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52016.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52016.log.gz 2024-09-12T16:36:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55799.log 2024-09-12T16:36:29.819 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58539.log: gzip 87.5% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65055.log 2024-09-12T16:36:29.819 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58539.log.gz 2024-09-12T16:36:29.820 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55799.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55799.log.gz 2024-09-12T16:36:29.831 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69938.log 2024-09-12T16:36:29.832 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65055.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65055.log.gz 2024-09-12T16:36:29.846 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68250.log 2024-09-12T16:36:29.847 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69938.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69938.log.gz 2024-09-12T16:36:29.855 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50120.log 2024-09-12T16:36:29.865 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68250.log: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64593.log 2024-09-12T16:36:29.866 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50120.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50120.log.gz 2024-09-12T16:36:29.866 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43007.log.gz 2024-09-12T16:36:29.868 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68250.log.gz 2024-09-12T16:36:29.873 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48104.log.gz 2024-09-12T16:36:29.874 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66938.log 2024-09-12T16:36:29.877 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64593.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64593.log.gz 2024-09-12T16:36:29.877 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73260.log 2024-09-12T16:36:29.880 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66938.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66938.log.gz 2024-09-12T16:36:29.880 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67396.log 2024-09-12T16:36:29.881 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73260.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73260.log.gz 2024-09-12T16:36:29.882 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72556.log 2024-09-12T16:36:29.883 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67396.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67396.log.gz 2024-09-12T16:36:29.884 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51058.log 2024-09-12T16:36:29.884 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72556.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72556.log.gz 2024-09-12T16:36:29.885 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57288.log 2024-09-12T16:36:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51058.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51058.log.gz 2024-09-12T16:36:29.886 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54841.log 2024-09-12T16:36:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57288.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57288.log.gz 2024-09-12T16:36:29.888 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79132.log 2024-09-12T16:36:29.891 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54841.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54841.log.gz 2024-09-12T16:36:29.891 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62804.log 2024-09-12T16:36:29.891 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79132.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79132.log.gz 2024-09-12T16:36:29.892 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51288.log 2024-09-12T16:36:29.893 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62804.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62804.log.gz 2024-09-12T16:36:29.893 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57987.log 2024-09-12T16:36:29.895 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51288.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51288.log.gz 2024-09-12T16:36:29.895 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48776.log 2024-09-12T16:36:29.897 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57987.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57987.log.gz 2024-09-12T16:36:29.897 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49290.log 2024-09-12T16:36:29.899 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48776.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48776.log.gz 2024-09-12T16:36:29.899 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81988.log 2024-09-12T16:36:29.899 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49290.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49290.log.gz 2024-09-12T16:36:29.901 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65505.log 2024-09-12T16:36:29.901 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81988.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81988.log.gz 2024-09-12T16:36:29.901 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65802.log 2024-09-12T16:36:29.903 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.65505.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65505.log.gz 2024-09-12T16:36:29.903 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47419.log 2024-09-12T16:36:29.904 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65802.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65802.log.gz 2024-09-12T16:36:29.905 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54953.log 2024-09-12T16:36:29.905 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47419.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47419.log.gz 2024-09-12T16:36:29.907 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.49601.log 2024-09-12T16:36:29.907 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54953.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54953.log.gz 2024-09-12T16:36:29.907 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56584.log 2024-09-12T16:36:29.909 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.1.49601.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.49601.log.gz 2024-09-12T16:36:29.910 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69218.log 2024-09-12T16:36:29.912 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56584.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56584.log.gz 2024-09-12T16:36:29.912 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74342.log 2024-09-12T16:36:29.914 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69218.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69218.log.gz 2024-09-12T16:36:29.914 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61759.log 2024-09-12T16:36:29.914 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74342.log: 79.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74342.log.gz 2024-09-12T16:36:29.914 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41533.log 2024-09-12T16:36:29.917 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61759.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61759.log.gz 2024-09-12T16:36:29.917 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57216.log 2024-09-12T16:36:29.919 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41533.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41533.log.gz 2024-09-12T16:36:29.919 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38002.log 2024-09-12T16:36:29.920 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57216.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57216.log.gz 2024-09-12T16:36:29.921 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60067.log 2024-09-12T16:36:29.921 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.38002.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38002.log.gz 2024-09-12T16:36:29.923 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43347.log 2024-09-12T16:36:29.923 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60067.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60067.log.gz 2024-09-12T16:36:29.923 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43452.log 2024-09-12T16:36:29.929 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43347.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43347.log.gz 2024-09-12T16:36:29.929 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56351.log 2024-09-12T16:36:29.931 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.43452.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.43452.log.gz 2024-09-12T16:36:29.931 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46660.log 2024-09-12T16:36:29.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56351.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56351.log.gz 2024-09-12T16:36:29.932 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42041.log 2024-09-12T16:36:29.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46660.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46660.log.gz 2024-09-12T16:36:29.934 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46680.log 2024-09-12T16:36:29.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.42041.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42041.log.gz 2024-09-12T16:36:29.936 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40851.log 2024-09-12T16:36:29.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46680.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46680.log.gz 2024-09-12T16:36:29.937 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45618.log 2024-09-12T16:36:29.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40851.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40851.log.gz 2024-09-12T16:36:29.940 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49349.log 2024-09-12T16:36:29.946 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.45618.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45618.log.gz 2024-09-12T16:36:29.946 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58129.log 2024-09-12T16:36:29.957 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49349.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43924.log.gz 2024-09-12T16:36:29.958 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46365.log 2024-09-12T16:36:29.960 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58129.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58129.log.gz 2024-09-12T16:36:29.960 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46179.log 2024-09-12T16:36:29.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.46365.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46365.log.gz 2024-09-12T16:36:29.961 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80140.log 2024-09-12T16:36:29.962 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46179.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46179.log.gz 2024-09-12T16:36:29.964 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59551.log 2024-09-12T16:36:29.964 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80140.log.gz 2024-09-12T16:36:29.964 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60778.log 2024-09-12T16:36:29.965 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59551.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59551.log.gz 2024-09-12T16:36:29.966 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43479.log 2024-09-12T16:36:29.967 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60778.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60778.log.gz 2024-09-12T16:36:29.967 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47183.log 2024-09-12T16:36:29.968 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.43479.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43479.log.gz 2024-09-12T16:36:29.968 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44264.log 2024-09-12T16:36:29.982 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66256.log 2024-09-12T16:36:29.986 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44264.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47262.log.gz 2024-09-12T16:36:29.988 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44264.log.gz 2024-09-12T16:36:29.989 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69038.log 2024-09-12T16:36:29.993 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66256.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66256.log.gz 2024-09-12T16:36:29.993 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48951.log 2024-09-12T16:36:29.994 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69038.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69038.log.gz 2024-09-12T16:36:29.994 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58153.log 2024-09-12T16:36:29.995 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48951.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48951.log.gz 2024-09-12T16:36:29.996 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57517.log 2024-09-12T16:36:29.997 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58153.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58153.log.gz 2024-09-12T16:36:29.997 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54796.log 2024-09-12T16:36:29.999 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57517.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57517.log.gz 2024-09-12T16:36:29.999 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62075.log 2024-09-12T16:36:30.000 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54796.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54796.log.gz 2024-09-12T16:36:30.000 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44156.log 2024-09-12T16:36:30.003 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62075.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62075.log.gz 2024-09-12T16:36:30.003 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43191.log 2024-09-12T16:36:30.004 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44156.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44156.log.gz 2024-09-12T16:36:30.005 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72105.log 2024-09-12T16:36:30.006 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43191.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43191.log.gz 2024-09-12T16:36:30.006 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60708.log 2024-09-12T16:36:30.008 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72105.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72105.log.gz 2024-09-12T16:36:30.008 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51724.log 2024-09-12T16:36:30.011 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60708.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60708.log.gz 2024-09-12T16:36:30.011 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70140.log 2024-09-12T16:36:30.011 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51724.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51724.log.gz 2024-09-12T16:36:30.011 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46975.log 2024-09-12T16:36:30.013 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70140.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70140.log.gz 2024-09-12T16:36:30.013 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78880.log 2024-09-12T16:36:30.015 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46975.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46975.log.gz 2024-09-12T16:36:30.015 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76191.log 2024-09-12T16:36:30.016 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78880.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78880.log.gz 2024-09-12T16:36:30.016 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55869.log 2024-09-12T16:36:30.016 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76191.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76191.log.gz 2024-09-12T16:36:30.019 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51021.log 2024-09-12T16:36:30.019 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55869.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55869.log.gz 2024-09-12T16:36:30.019 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52739.log 2024-09-12T16:36:30.019 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51021.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51021.log.gz 2024-09-12T16:36:30.019 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44623.log 2024-09-12T16:36:30.022 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52739.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52739.log.gz 2024-09-12T16:36:30.022 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72877.log 2024-09-12T16:36:30.023 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44623.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44623.log.gz 2024-09-12T16:36:30.023 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53622.log 2024-09-12T16:36:30.024 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72877.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72877.log.gz 2024-09-12T16:36:30.024 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51466.log 2024-09-12T16:36:30.026 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53622.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53622.log.gz 2024-09-12T16:36:30.026 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49428.log 2024-09-12T16:36:30.027 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51466.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51466.log.gz 2024-09-12T16:36:30.028 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38889.log 2024-09-12T16:36:30.040 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66566.log 2024-09-12T16:36:30.040 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.38889.log: 73.7% -- replaced with /var/log/ceph/cluster2-client.admin.38889.log.gz 2024-09-12T16:36:30.053 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45813.log 2024-09-12T16:36:30.056 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66566.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66566.log.gz 2024-09-12T16:36:30.059 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67747.log 2024-09-12T16:36:30.073 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45813.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44060.log 2024-09-12T16:36:30.073 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45813.log.gz 2024-09-12T16:36:30.083 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67747.log: 89.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69780.log 2024-09-12T16:36:30.083 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44060.log: 89.3% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44060.log.gz 2024-09-12T16:36:30.084 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74597.log.gz 2024-09-12T16:36:30.084 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55507.log 2024-09-12T16:36:30.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69780.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69780.log.gz 2024-09-12T16:36:30.086 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51703.log 2024-09-12T16:36:30.086 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55507.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55507.log.gz 2024-09-12T16:36:30.089 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63176.log 2024-09-12T16:36:30.089 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51703.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51703.log.gz 2024-09-12T16:36:30.089 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47220.log 2024-09-12T16:36:30.089 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63176.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63176.log.gz 2024-09-12T16:36:30.089 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67747.log.gz 2024-09-12T16:36:30.089 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58864.log 2024-09-12T16:36:30.091 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47220.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47220.log.gz 2024-09-12T16:36:30.092 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63034.log 2024-09-12T16:36:30.092 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58864.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58864.log.gz 2024-09-12T16:36:30.092 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51271.log 2024-09-12T16:36:30.095 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63034.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63034.log.gz 2024-09-12T16:36:30.095 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78165.log 2024-09-12T16:36:30.095 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51271.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51271.log.gz 2024-09-12T16:36:30.095 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49690.log 2024-09-12T16:36:30.095 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78165.log.gz 2024-09-12T16:36:30.098 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47535.log 2024-09-12T16:36:30.098 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49690.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49690.log.gz 2024-09-12T16:36:30.098 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73062.log 2024-09-12T16:36:30.098 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47535.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47535.log.gz 2024-09-12T16:36:30.098 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69578.log 2024-09-12T16:36:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73062.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73062.log.gz 2024-09-12T16:36:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55777.log 2024-09-12T16:36:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69578.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69578.log.gz 2024-09-12T16:36:30.101 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50041.log 2024-09-12T16:36:30.103 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55777.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55777.log.gz 2024-09-12T16:36:30.103 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66277.log 2024-09-12T16:36:30.105 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50041.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50041.log.gz 2024-09-12T16:36:30.105 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67644.log 2024-09-12T16:36:30.107 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66277.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66277.log.gz 2024-09-12T16:36:30.107 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53172.log 2024-09-12T16:36:30.108 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67644.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67644.log.gz 2024-09-12T16:36:30.108 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73387.log 2024-09-12T16:36:30.109 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53172.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53172.log.gz 2024-09-12T16:36:30.110 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43748.log 2024-09-12T16:36:30.113 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73387.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73387.log.gz 2024-09-12T16:36:30.113 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69870.log 2024-09-12T16:36:30.115 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43748.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43748.log.gz 2024-09-12T16:36:30.115 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53654.log 2024-09-12T16:36:30.116 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69870.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69870.log.gz 2024-09-12T16:36:30.116 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77619.log 2024-09-12T16:36:30.118 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53654.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53654.log.gz 2024-09-12T16:36:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40553.log 2024-09-12T16:36:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77619.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77619.log.gz 2024-09-12T16:36:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74236.log 2024-09-12T16:36:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40553.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40553.log.gz 2024-09-12T16:36:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59180.log 2024-09-12T16:36:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74236.log: 82.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74236.log.gz 2024-09-12T16:36:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74092.log 2024-09-12T16:36:30.122 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59180.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59180.log.gz 2024-09-12T16:36:30.122 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49936.log 2024-09-12T16:36:30.124 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74092.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74092.log.gz 2024-09-12T16:36:30.124 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61210.log 2024-09-12T16:36:30.126 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49936.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49936.log.gz 2024-09-12T16:36:30.126 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62488.log 2024-09-12T16:36:30.127 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61210.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61210.log.gz 2024-09-12T16:36:30.127 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73346.log 2024-09-12T16:36:30.129 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62488.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62488.log.gz 2024-09-12T16:36:30.129 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51308.log 2024-09-12T16:36:30.129 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73346.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47183.log.gz 2024-09-12T16:36:30.129 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73346.log.gz 2024-09-12T16:36:30.129 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68610.log 2024-09-12T16:36:30.140 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.51308.log: -- /var/log/ceph/cluster2-client.admin.37200.log 2024-09-12T16:36:30.140 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51308.log.gz 2024-09-12T16:36:30.140 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68610.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54471.log 2024-09-12T16:36:30.141 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68610.log.gz 2024-09-12T16:36:30.141 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37200.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37200.log.gz 2024-09-12T16:36:30.141 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27709.log 2024-09-12T16:36:30.141 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.54471.log: /var/log/ceph/cluster1-client.mirror.73992.log 2024-09-12T16:36:30.141 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54471.log.gz 2024-09-12T16:36:30.141 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.27709.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27709.log.gz 2024-09-12T16:36:30.141 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79510.log 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65468.log 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73992.log.gz 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79510.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53784.log 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65468.log: -- replaced with /var/log/ceph/cluster2-client.admin.79510.log.gz 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78754.log 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53784.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53784.log.gz 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56194.log 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78754.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78754.log.gz 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57924.log 2024-09-12T16:36:30.142 INFO:teuthology.orchestra.run.smithi019.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65468.log.gz 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59503.log 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.57924.log: -- replaced with /var/log/ceph/cluster1-client.mirror.56194.log.gz 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 87.6% --verbose -- /var/log/ceph/cluster2-client.admin.76653.log 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57924.log.gz 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38259.log 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59503.log: /var/log/ceph/cluster2-client.admin.76653.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76653.log.gz 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59503.log.gz 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.43659.log 2024-09-12T16:36:30.143 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.0.38259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58105.log 2024-09-12T16:36:30.161 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.2.43659.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.43659.log.gz 2024-09-12T16:36:30.161 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41670.log 2024-09-12T16:36:30.161 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58105.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58105.log.gz 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48892.log 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41670.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38259.log.gz 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66174.log 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48892.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48892.log.gz 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68498.log 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66174.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66174.log.gz 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53941.log 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68498.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68498.log.gz 2024-09-12T16:36:30.162 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61818.log 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53941.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53941.log.gz 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55688.log 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61818.log: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61818.log.gz 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46849.log 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55688.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55688.log.gz 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41670.log.gz 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51975.log 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71594.log 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46849.log: /var/log/ceph/cluster1-client.mirror.51975.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51975.log.gz 2024-09-12T16:36:30.163 INFO:teuthology.orchestra.run.smithi019.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46849.log.gz 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50584.log 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52038.log 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71594.log.gz 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50584.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50584.log.gz 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54335.log 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55391.log 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54335.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52038.log.gz 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46059.log 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55391.log: 89.8% 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54335.log.gz 2024-09-12T16:36:30.164 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55391.log.gz 2024-09-12T16:36:30.165 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38152.log 2024-09-12T16:36:30.165 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.46059.log: -- /var/log/ceph/cluster2-client.mirror.66980.log 2024-09-12T16:36:30.165 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38152.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46059.log.gz 2024-09-12T16:36:30.165 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72170.log 2024-09-12T16:36:30.165 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66980.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66980.log.gz 2024-09-12T16:36:30.165 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80602.log 2024-09-12T16:36:30.165 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72170.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72170.log.gz 2024-09-12T16:36:30.174 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56375.log 2024-09-12T16:36:30.174 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80602.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80602.log.gz 2024-09-12T16:36:30.174 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41557.log 2024-09-12T16:36:30.174 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56375.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56375.log.gz 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40601.log 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41557.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41557.log.gz 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53438.log 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40601.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40601.log.gz 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74034.log 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53438.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53438.log.gz 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72356.log 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74034.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74034.log.gz 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68318.log 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72356.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49349.log.gz 2024-09-12T16:36:30.175 INFO:teuthology.orchestra.run.smithi019.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72356.log.gz 2024-09-12T16:36:30.176 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42485.log 2024-09-12T16:36:30.176 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50700.log 2024-09-12T16:36:30.176 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68318.log.gz 2024-09-12T16:36:30.176 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43998.log 2024-09-12T16:36:30.176 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50700.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50700.log.gz 2024-09-12T16:36:30.176 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58057.log 2024-09-12T16:36:30.176 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43998.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42485.log.gz 2024-09-12T16:36:30.180 INFO:teuthology.orchestra.run.smithi019.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43998.log.gz 2024-09-12T16:36:30.180 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61550.log 2024-09-12T16:36:30.180 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52494.log 2024-09-12T16:36:30.181 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58057.log.gz 2024-09-12T16:36:30.181 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45684.log 2024-09-12T16:36:30.181 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.52494.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61550.log.gz 2024-09-12T16:36:30.181 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52494.log.gz 2024-09-12T16:36:30.181 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68295.log 2024-09-12T16:36:30.181 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42130.log 2024-09-12T16:36:30.181 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68295.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68295.log.gz 2024-09-12T16:36:30.182 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73237.log 2024-09-12T16:36:30.182 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42130.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42130.log.gz 2024-09-12T16:36:30.182 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81694.log 2024-09-12T16:36:30.183 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73237.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73237.log.gz 2024-09-12T16:36:30.183 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82618.log 2024-09-12T16:36:30.184 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81694.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81694.log.gz 2024-09-12T16:36:30.185 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69285.log 2024-09-12T16:36:30.185 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82618.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82618.log.gz 2024-09-12T16:36:30.185 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57698.log 2024-09-12T16:36:30.192 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69285.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69285.log.gz 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73281.log 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45684.log.gz 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71892.log 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57698.log.gz/var/log/ceph/cluster2-client.mirror.73281.log: 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64552.log 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71892.log: 88.3% 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73281.log.gz 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71892.log.gz 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60115.log 2024-09-12T16:36:30.193 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76233.log 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6%/var/log/ceph/cluster1-client.mirror.60115.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64552.log.gz 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60115.log.gz 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65864.log 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76233.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76233.log.gz 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67292.log 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71615.log 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65864.log: /var/log/ceph/cluster2-client.mirror.67292.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65864.log.gz 2024-09-12T16:36:30.194 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69983.log 2024-09-12T16:36:30.210 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.71615.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71615.log.gz 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67292.log.gz 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65719.log 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46159.log 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65719.log: /var/log/ceph/cluster1-client.mirror.69983.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69983.log.gz 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78670.log 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65719.log.gz 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60413.log 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78670.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78670.log.gz 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46159.log.gz 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60850.log 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46019.log 2024-09-12T16:36:30.211 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60850.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60850.log.gz 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43215.log 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46019.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60413.log.gz 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46019.log.gz 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/cluster2-client.mirror.51387.log 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59001.log 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43215.log: /var/log/ceph/cluster2-client.mirror.51387.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43215.log.gz 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51387.log.gz 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69623.log 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43796.log 2024-09-12T16:36:30.212 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69623.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69623.log.gz 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59001.log.gz 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41896.log 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48855.log 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.41896.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43796.log.gz 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr: 81.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41896.log.gz 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66690.log 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63360.log 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48855.log.gz 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47946.log 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63360.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63360.log.gz 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54377.log 2024-09-12T16:36:30.213 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47946.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66690.log.gz 2024-09-12T16:36:30.228 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51192.log 2024-09-12T16:36:30.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.54377.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54377.log.gz 2024-09-12T16:36:30.231 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46643.log 2024-09-12T16:36:30.231 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51192.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51192.log.gz 2024-09-12T16:36:30.235 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40649.log 2024-09-12T16:36:30.235 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46643.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46643.log.gz 2024-09-12T16:36:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79594.log 2024-09-12T16:36:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40649.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40649.log.gz 2024-09-12T16:36:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49050.log 2024-09-12T16:36:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79594.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79594.log.gz 2024-09-12T16:36:30.236 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80770.log 2024-09-12T16:36:30.239 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49050.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49050.log.gz 2024-09-12T16:36:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52449.log 2024-09-12T16:36:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80770.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80770.log.gz 2024-09-12T16:36:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.61818.log 2024-09-12T16:36:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52449.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52449.log.gz 2024-09-12T16:36:30.240 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51488.log 2024-09-12T16:36:30.244 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.1.61818.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.1.61818.log.gz 2024-09-12T16:36:30.244 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74662.log 2024-09-12T16:36:30.260 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57787.log 2024-09-12T16:36:30.262 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74662.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74662.log.gz 2024-09-12T16:36:30.267 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67664.log 2024-09-12T16:36:30.280 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57787.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57787.log.gz 2024-09-12T16:36:30.281 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62396.log 2024-09-12T16:36:30.283 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67664.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67664.log.gz 2024-09-12T16:36:30.290 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64299.log 2024-09-12T16:36:30.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62396.log: 89.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59409.log 2024-09-12T16:36:30.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64299.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64299.log.gz 2024-09-12T16:36:30.303 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62396.log.gz 2024-09-12T16:36:30.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53829.log 2024-09-12T16:36:30.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59409.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59409.log.gz 2024-09-12T16:36:30.322 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46099.log 2024-09-12T16:36:30.324 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53829.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53829.log.gz 2024-09-12T16:36:30.334 INFO:teuthology.orchestra.run.smithi019.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38152.log.gz 2024-09-12T16:36:30.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73408.log 2024-09-12T16:36:30.336 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46099.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49428.log.gz 2024-09-12T16:36:30.337 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46099.log.gz 2024-09-12T16:36:30.337 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55278.log 2024-09-12T16:36:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51117.log 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73408.log.gz 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37834.log 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51117.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55278.log.gz 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46300.log 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6%/var/log/ceph/cluster1-client.admin.37834.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51117.log.gz 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37834.log.gz 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48914.log 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82492.log 2024-09-12T16:36:30.341 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46300.log: /var/log/ceph/cluster2-client.mirror.48914.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46300.log.gz 2024-09-12T16:36:30.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70163.log 2024-09-12T16:36:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82492.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82492.log.gz 2024-09-12T16:36:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65200.log 2024-09-12T16:36:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70163.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70163.log.gz 2024-09-12T16:36:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72633.log 2024-09-12T16:36:30.343 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65200.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65200.log.gz 2024-09-12T16:36:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75351.log 2024-09-12T16:36:30.344 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72633.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72633.log.gz 2024-09-12T16:36:30.345 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67021.log 2024-09-12T16:36:30.345 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75351.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75351.log.gz 2024-09-12T16:36:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53124.log 2024-09-12T16:36:30.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67021.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67021.log.gz 2024-09-12T16:36:30.347 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62693.log 2024-09-12T16:36:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53124.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53124.log.gz 2024-09-12T16:36:30.348 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41288.log 2024-09-12T16:36:30.349 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62693.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62693.log.gz 2024-09-12T16:36:30.349 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53418.log 2024-09-12T16:36:30.350 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.41288.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41288.log.gz 2024-09-12T16:36:30.350 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50621.log 2024-09-12T16:36:30.352 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53418.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53418.log.gz 2024-09-12T16:36:30.353 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43618.log 2024-09-12T16:36:30.354 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50621.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50621.log.gz 2024-09-12T16:36:30.354 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81904.log 2024-09-12T16:36:30.354 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43618.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43618.log.gz 2024-09-12T16:36:30.355 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40152.log 2024-09-12T16:36:30.356 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.81904.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81904.log.gz 2024-09-12T16:36:30.356 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74432.log 2024-09-12T16:36:30.357 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40152.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40152.log.gz 2024-09-12T16:36:30.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69308.log 2024-09-12T16:36:30.358 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74432.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74432.log.gz 2024-09-12T16:36:30.358 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58406.log 2024-09-12T16:36:30.359 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69308.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69308.log.gz 2024-09-12T16:36:30.359 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58334.log 2024-09-12T16:36:30.360 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58406.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58406.log.gz 2024-09-12T16:36:30.360 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63245.log 2024-09-12T16:36:30.361 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58334.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58334.log.gz 2024-09-12T16:36:30.361 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51038.log 2024-09-12T16:36:30.363 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63245.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63245.log.gz 2024-09-12T16:36:30.364 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54314.log 2024-09-12T16:36:30.365 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51038.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51038.log.gz 2024-09-12T16:36:30.365 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68160.log 2024-09-12T16:36:30.366 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54314.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54314.log.gz 2024-09-12T16:36:30.367 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62167.log 2024-09-12T16:36:30.367 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68160.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68160.log.gz 2024-09-12T16:36:30.368 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41043.log 2024-09-12T16:36:30.369 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62167.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62167.log.gz 2024-09-12T16:36:30.369 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60043.log 2024-09-12T16:36:30.370 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41043.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41043.log.gz 2024-09-12T16:36:30.371 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39965.log 2024-09-12T16:36:30.371 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60043.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60043.log.gz 2024-09-12T16:36:30.371 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77955.log 2024-09-12T16:36:30.372 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.39965.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39965.log.gz 2024-09-12T16:36:30.373 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59862.log 2024-09-12T16:36:30.373 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77955.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77955.log.gz 2024-09-12T16:36:30.374 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44575.log 2024-09-12T16:36:30.374 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59862.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59862.log.gz 2024-09-12T16:36:30.375 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43284.log 2024-09-12T16:36:30.375 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44575.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44575.log.gz 2024-09-12T16:36:30.376 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52265.log 2024-09-12T16:36:30.381 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43284.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43284.log.gz 2024-09-12T16:36:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63480.log 2024-09-12T16:36:30.384 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52265.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52265.log.gz 2024-09-12T16:36:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57746.log 2024-09-12T16:36:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63480.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63480.log.gz 2024-09-12T16:36:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47924.log 2024-09-12T16:36:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57746.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57746.log.gz 2024-09-12T16:36:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67830.log 2024-09-12T16:36:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47924.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47924.log.gz 2024-09-12T16:36:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50759.log 2024-09-12T16:36:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67830.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67830.log.gz 2024-09-12T16:36:30.389 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39067.log 2024-09-12T16:36:30.390 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50759.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50759.log.gz 2024-09-12T16:36:30.391 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38701.log 2024-09-12T16:36:30.391 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.39067.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39067.log.gz 2024-09-12T16:36:30.392 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63902.log 2024-09-12T16:36:30.392 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38701.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38701.log.gz 2024-09-12T16:36:30.393 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66918.log 2024-09-12T16:36:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63902.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63902.log.gz 2024-09-12T16:36:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65740.log 2024-09-12T16:36:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66918.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66918.log.gz 2024-09-12T16:36:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57586.log 2024-09-12T16:36:30.397 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65740.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65740.log.gz 2024-09-12T16:36:30.397 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64744.log 2024-09-12T16:36:30.398 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.57586.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57586.log.gz 2024-09-12T16:36:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50742.log 2024-09-12T16:36:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64744.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64744.log.gz 2024-09-12T16:36:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62440.log 2024-09-12T16:36:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50742.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50742.log.gz 2024-09-12T16:36:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54102.log 2024-09-12T16:36:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62440.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62440.log.gz 2024-09-12T16:36:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.48247.log 2024-09-12T16:36:30.404 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54102.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54102.log.gz 2024-09-12T16:36:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62854.log 2024-09-12T16:36:30.407 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.48247.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.48247.log.gz 2024-09-12T16:36:30.408 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80644.log 2024-09-12T16:36:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62854.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62854.log.gz 2024-09-12T16:36:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68678.log 2024-09-12T16:36:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.80644.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80644.log.gz 2024-09-12T16:36:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50357.log 2024-09-12T16:36:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68678.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68678.log.gz 2024-09-12T16:36:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52514.log 2024-09-12T16:36:30.412 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50357.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50357.log.gz 2024-09-12T16:36:30.412 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50542.log 2024-09-12T16:36:30.414 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52514.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52514.log.gz 2024-09-12T16:36:30.415 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64069.log 2024-09-12T16:36:30.415 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50542.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50542.log.gz 2024-09-12T16:36:30.416 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44953.log 2024-09-12T16:36:30.417 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64069.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64069.log.gz 2024-09-12T16:36:30.417 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64723.log 2024-09-12T16:36:30.423 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44953.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44953.log.gz 2024-09-12T16:36:30.424 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.82114.log 2024-09-12T16:36:30.426 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64723.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64723.log.gz 2024-09-12T16:36:30.427 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64703.log 2024-09-12T16:36:30.427 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.82114.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.82114.log.gz 2024-09-12T16:36:30.428 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68565.log 2024-09-12T16:36:30.428 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64703.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64703.log.gz 2024-09-12T16:36:30.429 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59910.log 2024-09-12T16:36:30.429 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68565.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68565.log.gz 2024-09-12T16:36:30.430 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41315.log 2024-09-12T16:36:30.430 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59910.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59910.log.gz 2024-09-12T16:36:30.431 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42794.log 2024-09-12T16:36:30.431 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.41315.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41315.log.gz 2024-09-12T16:36:30.432 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51927.log 2024-09-12T16:36:30.438 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42794.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42794.log.gz 2024-09-12T16:36:30.438 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76989.log 2024-09-12T16:36:30.439 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.51927.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51927.log.gz 2024-09-12T16:36:30.440 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64090.log 2024-09-12T16:36:30.440 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76989.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76989.log.gz 2024-09-12T16:36:30.440 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39030.log 2024-09-12T16:36:30.442 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64090.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64090.log.gz 2024-09-12T16:36:30.443 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65636.log 2024-09-12T16:36:30.443 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.39030.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39030.log.gz 2024-09-12T16:36:30.443 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41091.log 2024-09-12T16:36:30.446 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65636.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65636.log.gz 2024-09-12T16:36:30.447 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69420.log 2024-09-12T16:36:30.447 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41091.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41091.log.gz 2024-09-12T16:36:30.448 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71343.log 2024-09-12T16:36:30.448 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69420.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69420.log.gz 2024-09-12T16:36:30.449 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50937.log 2024-09-12T16:36:30.450 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71343.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71343.log.gz 2024-09-12T16:36:30.450 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66070.log 2024-09-12T16:36:30.450 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50937.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50937.log.gz 2024-09-12T16:36:30.451 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66132.log 2024-09-12T16:36:30.453 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66070.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66070.log.gz 2024-09-12T16:36:30.454 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74921.log 2024-09-12T16:36:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66132.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66132.log.gz 2024-09-12T16:36:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38718.log 2024-09-12T16:36:30.458 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74921.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74921.log.gz 2024-09-12T16:36:30.459 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64619.log 2024-09-12T16:36:30.459 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.38718.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38718.log.gz 2024-09-12T16:36:30.459 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46955.log 2024-09-12T16:36:30.460 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64619.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64619.log.gz 2024-09-12T16:36:30.460 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47807.log 2024-09-12T16:36:30.461 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46955.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46955.log.gz 2024-09-12T16:36:30.462 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42315.log 2024-09-12T16:36:30.463 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47807.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47807.log.gz 2024-09-12T16:36:30.464 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47161.log 2024-09-12T16:36:30.467 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42315.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47946.log.gz 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63791.log 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47161.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47161.log.gz 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44599.log 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63791.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63791.log.gz 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42315.log.gz 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78502.log 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56806.log 2024-09-12T16:36:30.476 INFO:teuthology.orchestra.run.smithi019.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44599.log.gz 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78502.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78502.log.gz 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65096.log 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38957.log 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56806.log: /var/log/ceph/cluster2-client.mirror.65096.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56806.log.gz 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36918.log 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38957.log: 68.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38957.log.gz 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43081.log 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.36918.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36918.log.gz 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65096.log.gz 2024-09-12T16:36:30.477 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58358.log 2024-09-12T16:36:30.478 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.43081.log: -- /var/log/ceph/cluster1-client.mirror.53550.log 2024-09-12T16:36:30.482 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58358.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58358.log.gz 2024-09-12T16:36:30.482 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43081.log.gz 2024-09-12T16:36:30.482 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:30.483 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68948.log 2024-09-12T16:36:30.483 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53550.log: /var/log/ceph/cluster2.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53550.log.gz 2024-09-12T16:36:30.483 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73840.log 2024-09-12T16:36:30.483 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68948.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68948.log.gz 2024-09-12T16:36:30.483 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44201.log 2024-09-12T16:36:30.483 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73840.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73840.log.gz 2024-09-12T16:36:30.483 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72674.log 2024-09-12T16:36:30.485 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.44201.log: 87.6% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-12T16:36:30.485 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43369.log 2024-09-12T16:36:30.489 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72674.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72674.log.gz 2024-09-12T16:36:30.489 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63200.log 2024-09-12T16:36:30.490 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.43369.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44201.log.gz 2024-09-12T16:36:30.490 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56098.log 2024-09-12T16:36:30.490 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43369.log.gz 2024-09-12T16:36:30.490 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.63200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62117.log 2024-09-12T16:36:30.490 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63200.log.gz 2024-09-12T16:36:30.490 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56098.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56098.log.gz 2024-09-12T16:36:30.519 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72394.log 2024-09-12T16:36:30.519 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.62117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47577.log 2024-09-12T16:36:30.519 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.72394.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72394.log.gz 2024-09-12T16:36:30.519 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62117.log.gz 2024-09-12T16:36:30.519 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60296.log 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.47577.log: /var/log/ceph/cluster2-client.admin.37036.log 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47577.log.gz 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73504.log 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.37036.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37036.log.gz 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60296.log.gz 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41378.log 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.73504.log: -- /var/log/ceph/cluster1-client.mirror.54037.log 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr: 85.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73504.log.gz 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.0.41378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68633.log 2024-09-12T16:36:30.520 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54037.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54037.log.gz 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61321.log 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68633.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68633.log.gz 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68430.log 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61321.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.41378.log.gz 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61321.log.gz 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46279.log 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.68430.log: /var/log/ceph/cluster2-client.mirror.59744.log 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68430.log.gz 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.46279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69735.log 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59744.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46279.log.gz 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78544.log 2024-09-12T16:36:30.521 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59744.log.gz 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69735.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65822.log 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78544.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69735.log.gz 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78544.log.gz 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71848.log 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.65822.log: -- /var/log/ceph/cluster2-client.mirror.61620.log 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71848.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71848.log.gz 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49854.log 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.61620.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61620.log.gz 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65822.log.gz 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55341.log 2024-09-12T16:36:30.522 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.49854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76317.log 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49854.log.gz 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67788.log 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55341.log.gz 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76317.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76317.log.gz 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70284.log 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55965.log 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.70284.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70284.log.gz 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51639.log 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.55965.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67788.log.gz 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55965.log.gz 2024-09-12T16:36:30.523 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55483.log 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48793.log 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51639.log: /var/log/ceph/cluster1-client.mirror.55483.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55483.log.gz 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51639.log.gz 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60666.log 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55070.log 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.60666.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48793.log.gz 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56987.log 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55070.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60666.log.gz 2024-09-12T16:36:30.524 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58563.log 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56987.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55070.log.gz 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56987.log.gz 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66029.log 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73084.log 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58563.log: /var/log/ceph/cluster2-client.mirror.66029.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58563.log.gz 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43837.log 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66029.log.gz/var/log/ceph/cluster2-client.mirror.73084.log: 2024-09-12T16:36:30.525 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65657.log 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.43837.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43837.log.gz 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36943.log 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65657.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65657.log.gz 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73696.log 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.36943.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.admin.36943.log.gz 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66876.log 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73696.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73696.log.gz 2024-09-12T16:36:30.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66298.log 2024-09-12T16:36:30.533 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66876.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66876.log.gz 2024-09-12T16:36:30.533 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54013.log 2024-09-12T16:36:30.535 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.66298.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66298.log.gz 2024-09-12T16:36:30.535 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74871.log 2024-09-12T16:36:30.537 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54013.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54013.log.gz 2024-09-12T16:36:30.538 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.38336.log 2024-09-12T16:36:30.538 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.74871.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74871.log.gz 2024-09-12T16:36:30.538 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50241.log 2024-09-12T16:36:30.546 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.3.38336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63699.log 2024-09-12T16:36:30.553 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.50241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71320.log 2024-09-12T16:36:30.553 INFO:teuthology.orchestra.run.smithi019.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50241.log.gz 2024-09-12T16:36:30.557 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63699.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.38336.log.gz 2024-09-12T16:36:30.558 INFO:teuthology.orchestra.run.smithi019.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63699.log.gz 2024-09-12T16:36:30.559 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60826.log 2024-09-12T16:36:30.561 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.71320.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71320.log.gz 2024-09-12T16:36:30.562 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64993.log 2024-09-12T16:36:30.562 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.60826.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60826.log.gz 2024-09-12T16:36:30.562 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58447.log 2024-09-12T16:36:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64993.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64993.log.gz 2024-09-12T16:36:30.563 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53325.log 2024-09-12T16:36:30.571 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58447.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58447.log.gz 2024-09-12T16:36:30.571 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51488.log.gz 2024-09-12T16:36:30.571 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65014.log 2024-09-12T16:36:30.571 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.71652.log 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3%/var/log/ceph/cluster2-client.mirror.65014.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53325.log.gz 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54218.log 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.3.71652.log: -- replaced with /var/log/ceph/cluster2-client.mirror.65014.log.gz 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46486.log 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54218.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54218.log.gz 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74992.log 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.46486.log: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46486.log.gz 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42272.log 2024-09-12T16:36:30.572 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.admin.74992.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74992.log.gz 2024-09-12T16:36:30.573 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45512.log 2024-09-12T16:36:30.574 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.42272.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.71652.log.gz 2024-09-12T16:36:30.574 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55941.log 2024-09-12T16:36:30.577 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45512.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42272.log.gz 2024-09-12T16:36:30.578 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68183.log 2024-09-12T16:36:30.578 INFO:teuthology.orchestra.run.smithi019.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45512.log.gz/var/log/ceph/cluster1-client.mirror.55941.log: 2024-09-12T16:36:30.598 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55941.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70028.log 2024-09-12T16:36:30.598 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:30.598 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68183.log: 87.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38769.log 2024-09-12T16:36:30.598 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68183.log.gz 2024-09-12T16:36:30.598 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69443.log 2024-09-12T16:36:30.598 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70028.log: /var/log/ceph/cluster1-client.mirror.38769.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38769.log.gz 2024-09-12T16:36:30.598 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70028.log.gz 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76905.log 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.69443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70185.log 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69443.log.gz 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.76905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50416.log 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70185.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76905.log.gz 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59322.log 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70185.log.gz 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78964.log 2024-09-12T16:36:30.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.50416.log: /var/log/ceph/cluster1-client.mirror.59322.log: 83.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50416.log.gz 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59322.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54820.log 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74134.log 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78964.log: /var/log/ceph/cluster1-client.mirror.54820.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78964.log.gz 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75309.log 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54820.log.gz 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47124.log 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.75309.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75309.log.gz 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78418.log 2024-09-12T16:36:30.600 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47124.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47124.log.gz 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73740.log 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.78418.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78418.log.gz 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54861.log 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.73740.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73740.log.gz 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77073.log 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54861.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54861.log.gz 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65365.log 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77073.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77073.log.gz 2024-09-12T16:36:30.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61982.log 2024-09-12T16:36:30.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.65365.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65365.log.gz 2024-09-12T16:36:30.602 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54061.log 2024-09-12T16:36:30.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.61982.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61982.log.gz 2024-09-12T16:36:30.602 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52650.log 2024-09-12T16:36:30.602 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.54061.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54061.log.gz 2024-09-12T16:36:30.607 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57565.log 2024-09-12T16:36:30.607 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.52650.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52650.log.gz 2024-09-12T16:36:30.607 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67582.log 2024-09-12T16:36:30.608 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.57565.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57565.log.gz 2024-09-12T16:36:30.608 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73948.log 2024-09-12T16:36:30.608 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67582.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67582.log.gz 2024-09-12T16:36:30.608 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73004.log 2024-09-12T16:36:30.608 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73948.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73948.log.gz 2024-09-12T16:36:30.608 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79846.log 2024-09-12T16:36:30.614 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.73004.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73004.log.gz 2024-09-12T16:36:30.614 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79762.log 2024-09-12T16:36:30.614 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79846.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79846.log.gz 2024-09-12T16:36:30.614 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58081.log 2024-09-12T16:36:30.614 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79762.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79762.log.gz 2024-09-12T16:36:30.614 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58310.log 2024-09-12T16:36:30.614 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.58081.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58081.log.gz 2024-09-12T16:36:30.615 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40263.log 2024-09-12T16:36:30.615 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.58310.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58310.log.gz 2024-09-12T16:36:30.615 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55849.log 2024-09-12T16:36:30.616 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.40263.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40263.log.gz 2024-09-12T16:36:30.616 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53695.log 2024-09-12T16:36:30.618 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55849.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55849.log.gz 2024-09-12T16:36:30.618 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47656.log 2024-09-12T16:36:30.618 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53695.log: 81.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53695.log.gz 2024-09-12T16:36:30.620 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56893.log 2024-09-12T16:36:30.620 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.47656.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47656.log.gz 2024-09-12T16:36:30.620 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67520.log 2024-09-12T16:36:30.622 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56893.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56893.log.gz 2024-09-12T16:36:30.622 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62324.log 2024-09-12T16:36:30.623 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67520.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67520.log.gz 2024-09-12T16:36:30.624 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53849.log 2024-09-12T16:36:30.626 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.62324.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62324.log.gz 2024-09-12T16:36:30.626 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40200.log 2024-09-12T16:36:30.629 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.53849.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53849.log.gz 2024-09-12T16:36:30.629 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38974.log 2024-09-12T16:36:30.632 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.40200.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40200.log.gz 2024-09-12T16:36:30.632 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63649.log 2024-09-12T16:36:30.632 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.38974.log: 68.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38974.log.gz 2024-09-12T16:36:30.632 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68228.log 2024-09-12T16:36:30.635 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.63649.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63649.log.gz 2024-09-12T16:36:30.635 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45598.log 2024-09-12T16:36:30.635 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.68228.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68228.log.gz 2024-09-12T16:36:30.635 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44712.log 2024-09-12T16:36:30.641 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45598.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45598.log.gz 2024-09-12T16:36:30.642 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64528.log 2024-09-12T16:36:30.648 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.44712.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44712.log.gz 2024-09-12T16:36:30.648 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36994.log 2024-09-12T16:36:30.651 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64528.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64528.log.gz 2024-09-12T16:36:30.651 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59045.log 2024-09-12T16:36:30.651 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.36994.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36994.log.gz 2024-09-12T16:36:30.652 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51330.log 2024-09-12T16:36:30.653 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.59045.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59045.log.gz 2024-09-12T16:36:30.653 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72335.log 2024-09-12T16:36:30.663 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.51330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56676.log 2024-09-12T16:36:30.664 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.72335.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72335.log.gz 2024-09-12T16:36:30.677 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64386.log 2024-09-12T16:36:30.679 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56676.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56676.log.gz 2024-09-12T16:36:30.679 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74112.log 2024-09-12T16:36:30.686 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.64386.log: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53921.log 2024-09-12T16:36:30.686 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64386.log.gz 2024-09-12T16:36:30.699 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.74112.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56604.log 2024-09-12T16:36:30.700 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.74112.log.gz 2024-09-12T16:36:30.700 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.53921.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53921.log.gz 2024-09-12T16:36:30.706 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41223.log 2024-09-12T16:36:30.719 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.56604.log: gzip -5 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56604.log.gz 2024-09-12T16:36:30.720 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.2.65505.log 2024-09-12T16:36:30.725 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.41223.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41223.log.gz 2024-09-12T16:36:30.738 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55411.log 2024-09-12T16:36:30.746 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.2.65505.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.65505.log.gz 2024-09-12T16:36:30.751 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73084.log.gz 2024-09-12T16:36:30.752 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56122.log 2024-09-12T16:36:30.753 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.55411.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55411.log.gz 2024-09-12T16:36:30.753 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64323.log 2024-09-12T16:36:30.754 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.56122.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56122.log.gz 2024-09-12T16:36:30.755 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77871.log 2024-09-12T16:36:30.755 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.64323.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64323.log.gz 2024-09-12T16:36:30.757 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45383.log 2024-09-12T16:36:30.757 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.77871.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77871.log.gz 2024-09-12T16:36:30.757 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48336.log 2024-09-12T16:36:30.762 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.45383.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45383.log.gz 2024-09-12T16:36:30.763 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67706.log 2024-09-12T16:36:30.764 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48336.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48336.log.gz 2024-09-12T16:36:30.764 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70118.log 2024-09-12T16:36:30.765 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.67706.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67706.log.gz 2024-09-12T16:36:30.765 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48541.log 2024-09-12T16:36:30.765 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster1-client.mirror.70118.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70118.log.gz 2024-09-12T16:36:30.767 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79720.log 2024-09-12T16:36:30.767 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.48541.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48541.log.gz 2024-09-12T16:36:30.767 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59794.log 2024-09-12T16:36:30.767 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.admin.79720.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79720.log.gz 2024-09-12T16:36:30.769 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/cluster2-client.mirror.59794.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59794.log.gz 2024-09-12T16:36:30.781 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48914.log.gz 2024-09-12T16:36:30.877 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51330.log.gz 2024-09-12T16:36:30.886 INFO:teuthology.orchestra.run.smithi019.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74134.log.gz 2024-09-12T16:36:30.914 INFO:teuthology.orchestra.run.smithi019.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-12T16:36:34.672 INFO:teuthology.orchestra.run.smithi016.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-12T16:36:35.255 INFO:teuthology.orchestra.run.smithi016.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-12T16:36:36.398 INFO:teuthology.orchestra.run.smithi016.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-12T16:36:36.400 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-12T16:36:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m10.157s 2024-09-12T16:36:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m28.272s 2024-09-12T16:36:36.401 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m1.010s 2024-09-12T16:37:17.137 INFO:teuthology.orchestra.run.smithi019.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-12T16:37:17.884 INFO:teuthology.orchestra.run.smithi019.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-12T16:37:31.692 INFO:teuthology.orchestra.run.smithi019.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-12T16:37:31.694 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-09-12T16:37:31.694 INFO:teuthology.orchestra.run.smithi019.stderr:real 1m5.416s 2024-09-12T16:37:31.694 INFO:teuthology.orchestra.run.smithi019.stderr:user 3m2.340s 2024-09-12T16:37:31.694 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m7.291s 2024-09-12T16:37:31.694 INFO:tasks.ceph:Archiving logs... 2024-09-12T16:37:31.696 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/remote/smithi016/log 2024-09-12T16:37:31.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T16:37:35.450 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/remote/smithi019/log 2024-09-12T16:37:35.451 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-12T16:38:01.731 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-12T16:38:01.745 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph.py", line 1990, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2867, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2803, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 2464, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1696, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_d61b08865d4d98738f45ff2e4806d7fb6d03eca1/qa/tasks/ceph_manager.py", line 1687, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-09-12T16:38:01.746 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-12T16:38:01.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T16:38:01.749 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-12T16:38:01.833 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-12T16:38:01.844 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-12T16:38:01.844 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T16:38:01.847 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-12T16:38:01.862 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-09-12T16:38:01.865 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-09-12T16:38:01.867 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T16:38:01.867 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-09-12T16:38:01.867 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 -221us[ -235us] +/- 62ms 2024-09-12T16:38:01.867 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 14 +120us[ +106us] +/- 72ms 2024-09-12T16:38:01.867 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +196us[ +182us] +/- 91ms 2024-09-12T16:38:01.867 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T16:38:01.870 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-12T16:38:01.870 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-09-12T16:38:01.870 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 -216us[ -202us] +/- 62ms 2024-09-12T16:38:01.870 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 +124us[ +138us] +/- 72ms 2024-09-12T16:38:01.870 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +193us[ +207us] +/- 91ms 2024-09-12T16:38:01.870 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-12T16:38:01.871 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-12T16:38:01.881 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-12T16:38:01.882 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-12T16:38:01.894 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-12T16:38:01.979 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-12T16:38:02.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T16:38:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T16:38:02.090 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-12T16:38:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T16:38:02.121 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-12T16:38:02.133 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-12T16:38:02.212 INFO:teuthology.task.internal:Duration was 3032.975230 seconds 2024-09-12T16:38:02.212 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-12T16:38:02.253 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-12T16:38:02.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T16:38:02.256 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-12T16:38:02.299 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T16:38:02.302 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-12T16:38:02.640 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-12T16:38:02.640 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-09-12T16:38:02.640 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T16:38:02.666 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-09-12T16:38:02.666 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-12T16:38:02.706 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-12T16:38:02.706 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T16:38:02.709 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-12T16:38:02.777 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-12T16:38:02.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T16:38:02.781 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-12T16:38:02.935 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-12T16:38:02.947 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-12T16:38:02.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T16:38:02.974 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-12T16:38:03.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-12T16:38:03.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T16:38:03.017 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-12T16:38:03.039 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-09-12T16:38:03.064 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-09-12T16:38:03.091 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T16:38:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T16:38:03.147 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-12T16:38:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-12T16:38:03.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-12T16:38:03.174 INFO:teuthology.task.internal:Transferring archived files... 2024-09-12T16:38:03.175 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/remote/smithi016 2024-09-12T16:38:03.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T16:38:03.221 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-12_15:21:08-rbd-wip-dis-testing-distro-default-smithi/7902659/remote/smithi019 2024-09-12T16:38:03.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-12T16:38:03.255 INFO:teuthology.task.internal:Removing archive directory... 2024-09-12T16:38:03.255 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T16:38:03.258 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-12T16:38:03.306 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-12T16:38:03.320 INFO:teuthology.task.internal:Not uploading archives. 2024-09-12T16:38:03.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-12T16:38:03.335 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-12T16:38:03.335 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T16:38:03.338 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-12T16:38:03.351 INFO:teuthology.orchestra.run.smithi016.stdout: 265341 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 16:38 /home/ubuntu/cephtest 2024-09-12T16:38:03.361 INFO:teuthology.orchestra.run.smithi019.stdout: 265568 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 12 16:38 /home/ubuntu/cephtest 2024-09-12T16:38:03.362 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-12T16:38:03.375 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-12T16:38:03.388 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi019 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-09-12T16:38:03.391 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 3032.9752299785614 failure_reason: '"2024-09-12T16:28:54.486300+0000 mon.a (mon.0) 1595 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log' flavor: default owner: scheduled_dis@teuthology status: fail success: false 2024-09-12T16:38:03.391 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-12T16:38:03.484 INFO:teuthology.run:FAIL